From adfa0c0fb4bceae96a16b63a74d5bbd75da8f565 Mon Sep 17 00:00:00 2001 From: Quarto GHA Workflow Runner Date: Tue, 28 May 2024 19:56:23 +0000 Subject: [PATCH] Built site for gh-pages --- .nojekyll | 2 +- index.html | 4 +- .../add-folks-to-2i2c-github-teams.html | 273 +- search.json | 4203 ++++++++--------- sitemap.xml | 320 +- when-to-cloud.html | 4 +- 6 files changed, 2466 insertions(+), 2340 deletions(-) diff --git a/.nojekyll b/.nojekyll index 7cdf805e..3de7ddc4 100644 --- a/.nojekyll +++ b/.nojekyll @@ -1 +1 @@ -f1b71f04 \ No newline at end of file +f8466a30 \ No newline at end of file diff --git a/index.html b/index.html index 6cb669de..b927b7d4 100644 --- a/index.html +++ b/index.html @@ -7,7 +7,7 @@ - + EarthData Cloud Cookbook - NASA Earthdata Cloud Cookbook @@ -545,20 +579,32 @@

On this page

Edit this page

View source

Report an issue

@@ -582,15 +628,32 @@

How to Add Folks to the 2i2c Hub

-

We use GitHub Teams to manage access to the 2i2c Hub. To accomplish this, we use a dedicated GitHub Organization - nasa-openscapes-workshops - with GitHub Teams for admin (people who can add others to the Hub), workshop participants, Champions Cohorts, and long term access.

-

People are given Openscapes 2i2c JupyterHub access for workshops run by NASA Openscapes Mentors. This is the workflow to add individual users one by one. When we have a list of workshop participants, there are quicker ways to add all at once; we’ll document that after testing.

-
-

1. Share Google Form with participants

-

Ask participants: Please add your GitHub Username to this Google form for NASA Openscapes JupyterHub Access.

+ +

We use GitHub Teams to manage access to the 2i2c Hub. There are three different patterns of access:

+
    +
  1. Individually-managed long-term access for NASA-Openscapes mentors and others
    2. +
  2. Access for participants in a NASA-Openscapes champions cohort
    3. +
  3. Access for participants in a workshop
    4. +
+

Access for for cohorts and workshop participants currently uses the same workflow.

+

We are also working on a method to give just-in-time access for specific workshops without using GitHub Teams or other managed access. Details of this process will be added as we finalize it.

+
+

Individual long-term access

+

Individuals are given long-term access by adding them as members of the LongtermAccess-2i2c team in the NASA-Openscapes organization.

+

This is the workflow to add individual users one by one.

+
+

1. Share Google Form with users needing long-term access

+

Ask users: Please add your GitHub Username to this Google form for NASA Openscapes JupyterHub Access. Requests also sometimes come in via email directly from NASA mentors.

-
-

2. Google Form response setup

-

Set the form responses Openscapes_JupyterHubAccess to notify when there are responses. Also do this for any separate sheets used for workshop JupyterHub registration Click on Responses > click three vertical dots and then check “email notification.”

+
+

2. Google Form response setup

+

Set the form responses in Openscapes_JupyterHubAccess to notify when there are responses: Click on Responses > click three vertical dots and then check “email notification.”

alt text goes here

@@ -602,8 +665,8 @@

2. Google Form

-
-

3. Email alert of new signup

+
+

3. Email alert of new signup

When you get an email click ‘view response’ (not view summary)

@@ -611,20 +674,33 @@

3. Email alert o

-
-

4. Copy GitHub username

+
+

4. Copy GitHub username

On the response page, copy the Github Username alt text goes here

-
-

5. Add GitHub username to a GitHub Team

-

For workshop participants, go to the WorkshopAccess-2i2c Team in GitHub > Click Add member > Paste GitHub username. Note there are a separate GitHub Teams for Champions Cohorts and long term access.

+
+

5. Add GitHub username to the GitHub Team

+
+
+
+ +
+
+Tip +
+
+
+

If you are adding many new mentors to the long-term access team, you can use the same steps as for adding workshop participants

+
+
+

Go to the NASA Openscapes LongtermAccess-2i2c Team in GitHub > Click Add member > Paste GitHub username. Note there are separate GitHub Teams for Champions Cohorts and workshops.

alt text goes here

-

GitHub will ask you for 2-factor authentication for the first one you add in a batch, not after that.

-

People may see invitation notifications in two ways. They must accept the invitation to be able to access the Hub. This is what it looks like in their email inbox:

+

GitHub may ask you for 2-factor authentication for the first one you add in a batch, not after that.

+

People may see invitation notifications in two ways; in GitHub itself, and/or via email. They must accept the invitation to be able to access the Hub. This is what it looks like in their email inbox:

alt text goes here

@@ -642,40 +718,27 @@

5. Ad

-
-

6. Email confirmation

-

Go back to the form and grab their email address. Send one of the following emails to the person.

-

Email for workshop specific: 

-
-

Subject: NASA Openscapes Access to 2i2c Hub
-Hi [FIRST NAME],

-

I have added you to the NASA Openscapes GitHub Organization’s Workshop Team. Please check your email and accept this invitation from GitHub. Once you have joined the team, you will be able to access the JupyterHub.
-- Here is the link to the hub: https://openscapes.2i2c.cloud/
-- Here is the material for the Workshop NAME> https://nasa-openscapes.github.io/2023-ssc/.

-

Have a great workshop,

-
-

…

+
+

6. Email confirmation

+

Go back to the Google form response and grab their email address. Send the following emails to the person.

Email for General requests: 

-

Subject: NASA Openscapes Access to 2i2c Hub Hi [FIRST NAME],

+

Subject: NASA Openscapes Access to 2i2c Hub

+

Hi [FIRST NAME],

I have added you to the NASA Openscapes 2i2c Jupyter Hub. Here is the link to the hub: https://openscapes.2i2c.cloud/ 

There is a getting started guide in the NASA Earthdata Cloud Cookbook here: https://nasa-openscapes.github.io/earthdata-cloud-cookbook/

-

We’d love to know about the kind of work you are doing on the hub, and I’ll follow up in the next few months. 

-
-
+

We’d love to know about the kind of work you are doing on the hub. Please add a brief description of your progress as you go at https://github.com/NASA-Openscapes/2i2cAccessPolicies/discussions/2. We will follow up in the next few months. 

Best,

-

… 

-

 

-
-

7. Register date in Form Response Sheet

-

In the Form Response Sheet, in Column G add the date they were added to the Hub - in part so that we as admins knew if someone had already fulfilled the request. Rows 453+ are the most recent, but it can vary. Check Column A for the date of the form request.

+
+

7. Register date in Form Response Sheet

+

In the Form Response Sheet, in column Added to NASA 2i2c hub add the date they were added to the Hub, or if the request came via email add a new row - in part so that we as admins knew if someone had already fulfilled the request. Check Column A for the date of the form request.

-
-

Directions for workshop participants

+
+

Directions for invitees

    -

  1. Please go to https://github.com/nasa-openscapes-workshops

    2. +

  2. Please go to https://github.com/nasa-openscapes

  3. Click the big yellow bar that says “you’ve been invited” 

  4. Not seeing that bar? 

      @@ -685,11 +748,123 @@

      Direc

+
+
+

Adding workshop participants or Champions cohorts as a batch

+

Participants in workshops run by NASA Openscapes Mentors are given Openscapes 2i2c JupyterHub access, as are participants in Champions Cohorts.

+

We use a dedicated GitHub Organization - nasa-openscapes-workshops - to manage access, with GitHub Teams for workshops and Champions Cohorts.

+
+

1. Create a team in nasa-openscapes-workshops

+

There are several teams in this organization; the AdminTeam team is for members who have permission to create teams and add members to teams.

+
    +
  • If this is for a one-off workshop, name the team [workshop-name]-[workshop-date], with workshop date in the format yyyy-mm-dd
    • +
  • If this is for a new champions cohort, name the team nasa-champions-yyyy
    • +
+
+
+

2. Add team name to workshop registry

+

We keep a registry of all workshops in this sheet. Add the name of the GitHub team to 2i2c access type column in the appropriate row.

+
+
+

3. Add teams to 2i2c Hub configuration

+

Make a pull request to the Openscapes common.values.yaml file in the 2i2c infrastructure repository, adding the new team to the allowed-teams item in whichever sections are appropriate for your workshop/cohort (likely all of them):

+
    +
  • display_name: Python
    • +
  • display_name: R
    • +
  • display_name: Matlab
    • +
  • display_name: "Bring your own image"
    • +
+

These will determine which images are available to participants when they log in to the Hub.

+

Team specification is in the form [github-org]:[team] (e.g., nasa-openscapes-workshops:champions-access-2i2c-2024)

+

You also need to add the team to the allowed_organizations item under hub > config > GitHubOAuthenticator

+

You can see an example pull request here.

+
+
+

4. Share Google Form with participants needing access for the cohort or workshop

+

Ask users: Please add your GitHub Username to this Google form for NASA Openscapes JupyterHub Access.

+
+
+

5. Add users to a Team in a batch using kyber

+
+
+
+ +
+
+Tip +
+
+
+

Do this fairly close to the day of the workshop to try to get the greatest number particpants to have filled out the form, and reduce the number of manual additions on the day of the workshop.

+
+
+
    +
  1. Open the spreadsheet of form responses, select the GitHub usernames from the rows containing the people you want to add, and copy it to your clipboard.
    2. +
+
+
+
+ +
+
+Tip +
+
+
+

Some workshop instructors will use their own form to record participants, rather than the Openscapes form. In that case use those responses instead. But do paste all those rows into this Main form; it is important that we have a record of everyone using the Hub and a way to contact them via email.

+
+
+
    +
  1. In R, do the following:
    2. +
+
library(kyber)
+library(datapasta)
+
+# Run datapasta::tribble_paste() or use the RStudio Addin 'Paste as tribble'
+
+# assign the pasted output to a new object:
+responses <- tribble::tribble(
+    # The stuff that was pasted
+)
+

The responses object will have a column of usernames. It likely won’t be called username, so you can rename it or even easier, change the name inside the tribble::tribble() call that you pasted. Verify these are the usernames you want to add, and run kyber::add_team_members() as follows:

+
team_name <- "the name of the github team you want to add to"
+
+add_team_members(
+  team_name, 
+  members = responses$username, 
+  org = "nasa-openscapes-workshops"
+)
+
    +
  1. Go the appropriate team in the nasa-openscapes-workshops organization and verify that there are the appropriate “pending members”:
    2. +
+
+
+

alt text goes here

+
+
+
    +
  1. Go back to the spreadsheet of responses, and fill in the Added to NASA 2i2c hub column with the date you added them.
    2. +
+
+
+

Directions for workshop participants

+
    +

  1. Please go to https://github.com/nasa-openscapes-workshops

    2. +

  2. Click the big yellow bar that says “you’ve been invited” 

    3. +

  3. Not seeing that bar? 

    +
      +

    1. Are you logged into GitHub.com?

      2. +

    2. Have you shared your GitHub username via this form?

      3. +

    3. You can also check your email that you use for GitHub and look for an invitation from GitHub and NASA-Openscapes-Workshops

      4. +
    4. +
+
+

Troubleshooting

Users get 403 error

-

This is a common issue. User must find email notification from GitHub and accept invitation. If they had clicked in email to accept but did not complete accepting in GitHub, edit their invitation in GitHub to uninvite and reinvite. They should get new email to accept.

+

This is a common issue. User must find email notification from GitHub and accept invitation. If they had clicked in email to accept but did not complete accepting in GitHub, edit their invitation in GitHub to uninvite and reinvite. They should get a new email to accept.

alt text goes here

@@ -700,12 +875,12 @@

Users get 403 errorDidn’t get email

Go to https://github.com/nasa-openscapes-workshops, see the big yellow bar that says You’re invited. 

-
-
-

Notes on process and any issues

+
+

Notes on process and any issues

Yuvi has started documenting the issues we are running into here -> https://hackmd.io/tWfqVai4SDC1CbQ4mhfIbw 

+
diff --git a/search.json b/search.json index 2ef9c640..fd73b728 100644 --- a/search.json +++ b/search.json @@ -1,24 +1,255 @@ [ { - "objectID": "LICENSE.html", - "href": "LICENSE.html", - "title": "LICENSE", + "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html", + "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html", + "title": "GES DISC - MERRA2", "section": "", - "text": "Adapted from The Turing Way\nOpenscapes makes all of their materials publicly available under open source licenses.\nThe process documents and data are made available under a CC-BY license. Software are made available under an MIT license.\nThe license text listed below (describing both CC-BY and MIT licenses as well as their usage in Openscapes) is re-used under a CC-BY license from The Carpentries community materials. (Specifically from the Reproducible Science Curriculum).\n\n\nAll documentation and chapter materials in this repository are made available under the Creative Commons Attribution license. The following is a human-readable summary of (and not a substitute for) the full legal text of the CC BY 4.0 license.\nYou are free:\n\nto Share—copy and redistribute the material in any medium or format\nto Adapt—remix, transform, and build upon the material\n\nfor any purpose, even commercially.\nThe licensor cannot revoke these freedoms as long as you follow the license terms.\nUnder the following terms:\n\nAttribution—You must give appropriate credit (mentioning that your work is derived from work that is Copyright © Openscapes and, where practical, linking to https://github.com/openscapes/approach-guide), provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.\nNo additional restrictions—You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.\n\nWith the understanding that:\n\nYou do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation.\nNo warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.\n\n\n\n\nExcept where otherwise noted, the software and infrastructure provided by the The Turing Way Community are made available under the OSI-approved MIT license.\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\nTHE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE." + "text": "Many of NASA’s current and legacy data collections are archive in netCDF4 format. By itself, netCDF4 are not cloud optimized and reading these files can take as long from a personal/local work environment as it takes to read the data from a working environment deployed in the cloud. Using Kerchunk, we can treat these files as cloud optimized assets by creating metadata json file describing existing netCDF4 files, their chunks, and where to access them. The json reference files can be read in using Zarr and Xarray for efficient reads and fast processing." }, { - "objectID": "LICENSE.html#process-documents-and-data", - "href": "LICENSE.html#process-documents-and-data", - "title": "LICENSE", + "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#requirements", + "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#requirements", + "title": "GES DISC - MERRA2", + "section": "Requirements", + "text": "Requirements\n\n1. AWS instance running in us-west-2\nNASA Earthdata Cloud data in S3 can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region.\n\n\n2. Earthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\n3. netrc File\nYou will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. For additional information see: Authentication for NASA Earthdata." + }, + { + "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#import-required-packages", + "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#import-required-packages", + "title": "GES DISC - MERRA2", + "section": "Import required packages", + "text": "Import required packages\n\nimport requests\nimport xarray as xr\nimport ujson\nimport s3fs\nimport fsspec\nfrom tqdm import tqdm\nfrom glob import glob\nimport os\nimport pathlib\nimport hvplot.xarray\n\nfrom kerchunk.hdf import SingleHdf5ToZarr\nfrom kerchunk.combine import MultiZarrToZarr\n\n# The xarray produced from the reference file throws a SerializationWarning for each variable. Will need to explore why\nimport warnings\nwarnings.simplefilter(\"ignore\")" + }, + { + "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#create-dask-client-to-process-the-output-json-file-in-parallel", + "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#create-dask-client-to-process-the-output-json-file-in-parallel", + "title": "GES DISC - MERRA2", + "section": "Create Dask client to process the output json file in parallel", + "text": "Create Dask client to process the output json file in parallel\nGenerating the Kerchunk reference file can take some time depending on the internal structure of the data. Dask allows us to execute the reference file generation process in parallel, thus speeding up the overall process.\n\nimport dask\nfrom dask.distributed import Client\nclient = Client(n_workers=4)\nclient\n\n\n \n \n Client\n Client-3e6c0be3-d18c-11ec-809e-527eee20f3f0\n \n\n\n\nConnection method: Cluster object\nCluster type: distributed.LocalCluster\n\n\nDashboard: http://127.0.0.1:8787/status\n\n\n\n\n\nCluster Info\n\n\n\n\n\n\nLocalCluster\n64356e89\n\n\n\nDashboard: http://127.0.0.1:8787/status\nWorkers: 4\n\n\nTotal threads: 4\nTotal memory: 7.48 GiB\n\n\nStatus: running\nUsing processes: True\n\n\n\n\n\nScheduler Info\n\n\n\n\n\n\nScheduler\nScheduler-9811e6a9-be9d-4a54-9cc3-f17a8d5ea5bb\n\n\n\nComm: tcp://127.0.0.1:33947\nWorkers: 4\n\n\nDashboard: http://127.0.0.1:8787/status\nTotal threads: 4\n\n\nStarted: Just now\nTotal memory: 7.48 GiB\n\n\n\n\n\n\nWorkers\n\n\n\n\n\nWorker: 0\n\n\n\nComm: tcp://127.0.0.1:38069\nTotal threads: 1\n\n\nDashboard: http://127.0.0.1:44433/status\nMemory: 1.87 GiB\n\n\nNanny: tcp://127.0.0.1:43685\n\n\n\nLocal directory: /home/jovyan/earthdata-cloud-cookbook/examples/GESDISC/dask-worker-space/worker-6b5pkr_f\n\n\n\n\n\n\n\n\n\n\nWorker: 1\n\n\n\nComm: tcp://127.0.0.1:32859\nTotal threads: 1\n\n\nDashboard: http://127.0.0.1:36075/status\nMemory: 1.87 GiB\n\n\nNanny: tcp://127.0.0.1:35291\n\n\n\nLocal directory: /home/jovyan/earthdata-cloud-cookbook/examples/GESDISC/dask-worker-space/worker-piqb4cet\n\n\n\n\n\n\n\n\n\n\nWorker: 2\n\n\n\nComm: tcp://127.0.0.1:36601\nTotal threads: 1\n\n\nDashboard: http://127.0.0.1:45185/status\nMemory: 1.87 GiB\n\n\nNanny: tcp://127.0.0.1:41163\n\n\n\nLocal directory: /home/jovyan/earthdata-cloud-cookbook/examples/GESDISC/dask-worker-space/worker-d_91r19y\n\n\n\n\n\n\n\n\n\n\nWorker: 3\n\n\n\nComm: tcp://127.0.0.1:34309\nTotal threads: 1\n\n\nDashboard: http://127.0.0.1:38301/status\nMemory: 1.87 GiB\n\n\nNanny: tcp://127.0.0.1:44119\n\n\n\nLocal directory: /home/jovyan/earthdata-cloud-cookbook/examples/GESDISC/dask-worker-space/worker-9wa5m1vc" + }, + { + "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#get-temporary-s3-credentials", + "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#get-temporary-s3-credentials", + "title": "GES DISC - MERRA2", + "section": "Get temporary S3 credentials", + "text": "Get temporary S3 credentials\nTemporary S3 credentials need to be passed to AWS. Note, these credentials must be refreshed after 1 hour.\n\ns3_cred_endpoint = {\n 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n 'ornldaac':'https://data.ornldaac.earthdata.nasa.gov/s3credentials',\n 'gesdisc':'https://data.gesdisc.earthdata.nasa.gov/s3credentials'\n}\n\n\ndef get_temp_creds():\n temp_creds_url = s3_cred_endpoint['gesdisc']\n return requests.get(temp_creds_url).json()\n\n\ntemp_creds_req = get_temp_creds()" + }, + { + "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#direct-access-a-single-netcdf4-file", + "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#direct-access-a-single-netcdf4-file", + "title": "GES DISC - MERRA2", + "section": "Direct Access a single netCDF4 file", + "text": "Direct Access a single netCDF4 file\nPass temporary credentials to our filesystem object to access the S3 assets\n\nfs = s3fs.S3FileSystem(\n anon=False,\n key=temp_creds_req['accessKeyId'],\n secret=temp_creds_req['secretAccessKey'],\n token=temp_creds_req['sessionToken']\n)\n\n\nurl = 's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4'\n\n\ns3_file_obj = fs.open(url, mode='rb')\n\nTime how long it takes to directly access a cloud asset for comparisons later.\n\n%%time\n\nxr_ds = xr.open_dataset(s3_file_obj, chunks='auto', engine='h5netcdf')\nxr_ds\n\nCPU times: user 2.9 s, sys: 228 ms, total: 3.13 s\nWall time: 7.53 s\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (lon: 576, lat: 361, time: 24)\nCoordinates:\n * lon (lon) float64 -180.0 -179.4 -178.8 -178.1 ... 178.1 178.8 179.4\n * lat (lat) float64 -90.0 -89.5 -89.0 -88.5 ... 88.5 89.0 89.5 90.0\n * time (time) datetime64[ns] 2019-05-01T00:30:00 ... 2019-05-01T23:30:00\nData variables: (12/47)\n CLDPRS (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n CLDTMP (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n DISPH (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n H1000 (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n H250 (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n H500 (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n ... ...\n V250 (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n V2M (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n V500 (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n V50M (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n V850 (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n ZLCL (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\nAttributes: (12/30)\n History: Original file generated: Sat May 11 22...\n Comment: GMAO filename: d5124_m2_jan10.tavg1_2d...\n Filename: MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4\n Conventions: CF-1\n Institution: NASA Global Modeling and Assimilation ...\n References: http://gmao.gsfc.nasa.gov\n ... ...\n Contact: http://gmao.gsfc.nasa.gov\n identifier_product_doi: 10.5067/VJAFPLI1CSIV\n RangeBeginningDate: 2019-05-01\n RangeBeginningTime: 00:00:00.000000\n RangeEndingDate: 2019-05-01\n RangeEndingTime: 23:59:59.000000xarray.DatasetDimensions:lon: 576lat: 361time: 24Coordinates: (3)lon(lon)float64-180.0 -179.4 ... 178.8 179.4long_name :longitudeunits :degrees_eastvmax :1000000000000000.0vmin :-1000000000000000.0valid_range :[-1.e+15 1.e+15]array([-180. , -179.375, -178.75 , ..., 178.125, 178.75 , 179.375])lat(lat)float64-90.0 -89.5 -89.0 ... 89.5 90.0long_name :latitudeunits :degrees_northvmax :1000000000000000.0vmin :-1000000000000000.0valid_range :[-1.e+15 1.e+15]array([-90. , -89.5, -89. , ..., 89. , 89.5, 90. ])time(time)datetime64[ns]2019-05-01T00:30:00 ... 2019-05-...long_name :timetime_increment :10000begin_date :20190501begin_time :3000vmax :1000000000000000.0vmin :-1000000000000000.0valid_range :[-1.e+15 1.e+15]array(['2019-05-01T00:30:00.000000000', '2019-05-01T01:30:00.000000000',\n '2019-05-01T02:30:00.000000000', '2019-05-01T03:30:00.000000000',\n '2019-05-01T04:30:00.000000000', '2019-05-01T05:30:00.000000000',\n '2019-05-01T06:30:00.000000000', '2019-05-01T07:30:00.000000000',\n '2019-05-01T08:30:00.000000000', '2019-05-01T09:30:00.000000000',\n '2019-05-01T10:30:00.000000000', '2019-05-01T11:30:00.000000000',\n '2019-05-01T12:30:00.000000000', '2019-05-01T13:30:00.000000000',\n '2019-05-01T14:30:00.000000000', '2019-05-01T15:30:00.000000000',\n '2019-05-01T16:30:00.000000000', '2019-05-01T17:30:00.000000000',\n '2019-05-01T18:30:00.000000000', '2019-05-01T19:30:00.000000000',\n '2019-05-01T20:30:00.000000000', '2019-05-01T21:30:00.000000000',\n '2019-05-01T22:30:00.000000000', '2019-05-01T23:30:00.000000000'],\n dtype='datetime64[ns]')Data variables: (47)CLDPRS(time, lat, lon)float32dask.array<chunksize=(24, 361, 576), meta=np.ndarray>long_name :cloud_top_pressureunits :Pafmissing_value :1000000000000000.0standard_name :cloud_top_pressurevmax :1000000000000000.0vmin :-1000000000000000.0valid_range :[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nCLDTMP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ncloud_top_temperature\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ncloud_top_temperature\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nDISPH\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nzero_plane_displacement_height\n\nunits :\n\nm\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nzero_plane_displacement_height\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH1000\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nheight_at_1000_mb\n\nunits :\n\nm\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nheight_at_1000_mb\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nheight_at_250_hPa\n\nunits :\n\nm\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nheight_at_250_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nheight_at_500_hPa\n\nunits :\n\nm\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nheight_at_500_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nheight_at_850_hPa\n\nunits :\n\nm\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nheight_at_850_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nOMEGA500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nomega_at_500_hPa\n\nunits :\n\nPa s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nomega_at_500_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nPBLTOP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\npbltop_pressure\n\nunits :\n\nPa\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\npbltop_pressure\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nPS\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nsurface_pressure\n\nunits :\n\nPa\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nsurface_pressure\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nspecific_humidity_at_250_hPa\n\nunits :\n\nkg kg-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nspecific_humidity_at_250_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nspecific_humidity_at_500_hPa\n\nunits :\n\nkg kg-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nspecific_humidity_at_500_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nspecific_humidity_at_850_hPa\n\nunits :\n\nkg kg-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nspecific_humidity_at_850_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQV10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\n10-meter_specific_humidity\n\nunits :\n\nkg kg-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\n10-meter_specific_humidity\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQV2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\n2-meter_specific_humidity\n\nunits :\n\nkg kg-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\n2-meter_specific_humidity\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSLP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nsea_level_pressure\n\nunits :\n\nPa\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nsea_level_pressure\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\n10-meter_air_temperature\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\n10-meter_air_temperature\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nair_temperature_at_250_hPa\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nair_temperature_at_250_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\n2-meter_air_temperature\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\n2-meter_air_temperature\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2MDEW\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ndew_point_temperature_at_2_m\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ndew_point_temperature_at_2_m\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2MWET\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nwet_bulb_temperature_at_2_m\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nwet_bulb_temperature_at_2_m\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nair_temperature_at_500_hPa\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nair_temperature_at_500_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nair_temperature_at_850_hPa\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nair_temperature_at_850_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTO3\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntotal_column_ozone\n\nunits :\n\nDobsons\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntotal_column_ozone\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTOX\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntotal_column_odd_oxygen\n\nunits :\n\nkg m-2\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntotal_column_odd_oxygen\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQI\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntotal_precipitable_ice_water\n\nunits :\n\nkg m-2\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntotal_precipitable_ice_water\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQL\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntotal_precipitable_liquid_water\n\nunits :\n\nkg m-2\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntotal_precipitable_liquid_water\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQV\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntotal_precipitable_water_vapor\n\nunits :\n\nkg m-2\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntotal_precipitable_water_vapor\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPB\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntropopause_pressure_based_on_blended_estimate\n\nunits :\n\nPa\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntropopause_pressure_based_on_blended_estimate\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPT\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntropopause_pressure_based_on_thermal_estimate\n\nunits :\n\nPa\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntropopause_pressure_based_on_thermal_estimate\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPV\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntropopause_pressure_based_on_EPV_estimate\n\nunits :\n\nPa\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntropopause_pressure_based_on_EPV_estimate\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPQ\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntropopause_specific_humidity_using_blended_TROPP_estimate\n\nunits :\n\nkg kg-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntropopause_specific_humidity_using_blended_TROPP_estimate\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPT\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntropopause_temperature_using_blended_TROPP_estimate\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntropopause_temperature_using_blended_TROPP_estimate\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTS\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nsurface_skin_temperature\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nsurface_skin_temperature\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\n10-meter_eastward_wind\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\n10-meter_eastward_wind\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\neastward_wind_at_250_hPa\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\neastward_wind_at_250_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\n2-meter_eastward_wind\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\n2-meter_eastward_wind\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\neastward_wind_at_500_hPa\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\neastward_wind_at_500_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU50M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\neastward_wind_at_50_meters\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\neastward_wind_at_50_meters\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\neastward_wind_at_850_hPa\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\neastward_wind_at_850_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\n10-meter_northward_wind\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\n10-meter_northward_wind\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nnorthward_wind_at_250_hPa\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nnorthward_wind_at_250_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\n2-meter_northward_wind\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\n2-meter_northward_wind\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nnorthward_wind_at_500_hPa\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nnorthward_wind_at_500_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV50M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nnorthward_wind_at_50_meters\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nnorthward_wind_at_50_meters\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nnorthward_wind_at_850_hPa\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nnorthward_wind_at_850_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nZLCL\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nlifting_condensation_level\n\nunits :\n\nm\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nlifting_condensation_level\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nAttributes: (30)History :Original file generated: Sat May 11 22:08:52 2019 GMTComment :GMAO filename: d5124_m2_jan10.tavg1_2d_slv_Nx.20190501.nc4Filename :MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4Conventions :CF-1Institution :NASA Global Modeling and Assimilation OfficeReferences :http://gmao.gsfc.nasa.govFormat :NetCDF-4/HDF-5SpatialCoverage :globalVersionID :5.12.4TemporalRange :1980-01-01 -> 2016-12-31identifier_product_doi_authority :http://dx.doi.org/ShortName :M2T1NXSLVGranuleID :MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4ProductionDateTime :Original file generated: Sat May 11 22:08:52 2019 GMTLongName :MERRA2 tavg1_2d_slv_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Single-Level DiagnosticsTitle :MERRA2 tavg1_2d_slv_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Single-Level DiagnosticsSouthernmostLatitude :-90.0NorthernmostLatitude :90.0WesternmostLongitude :-180.0EasternmostLongitude :179.375LatitudeResolution :0.5LongitudeResolution :0.625DataResolution :0.5 x 0.625Source :CVS tag: GEOSadas-5_12_4_p16_sp3_M2-OPSContact :http://gmao.gsfc.nasa.govidentifier_product_doi :10.5067/VJAFPLI1CSIVRangeBeginningDate :2019-05-01RangeBeginningTime :00:00:00.000000RangeEndingDate :2019-05-01RangeEndingTime :23:59:59.000000" + }, + { + "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#specify-a-list-of-s3-urls", + "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#specify-a-list-of-s3-urls", + "title": "GES DISC - MERRA2", + "section": "Specify a list of S3 URLs", + "text": "Specify a list of S3 URLs\nData Collection: MERRA2_400.tavg1_2d_slv_Nx\nTime Range: 05/01/2019 - 05/31/2019\n\nurls = ['s3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190502.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190503.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190504.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190505.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190506.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190507.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190508.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190509.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190510.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190511.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190512.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190513.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190514.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190515.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190516.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190517.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190518.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190519.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190520.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190521.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190522.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190523.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190524.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190525.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190526.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190527.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190528.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190529.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190530.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190531.nc4']" + }, + { + "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#generate-the-kerchunk-reference-files.", + "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#generate-the-kerchunk-reference-files.", + "title": "GES DISC - MERRA2", + "section": "Generate the Kerchunk reference files.", + "text": "Generate the Kerchunk reference files.\nDefine a function to generate the Kerchunk reference files. These files can take a little time to generate.\n\ndef gen_json(u):\n so = dict(\n mode= \"rb\", \n anon= False, \n default_fill_cache= False,\n default_cache_type= \"none\"\n )\n with fs.open(u, **so) as infile:\n h5chunks = SingleHdf5ToZarr(infile, u, inline_threshold=300)\n with open(f\"jsons/{u.split('/')[-1]}.json\", 'wb') as outf:\n outf.write(ujson.dumps(h5chunks.translate()).encode())\n\nCreate output jsons directory if one does not exist.\n\npathlib.Path('./jsons/').mkdir(exist_ok=True)\n\nUse the Dask Delayed function to create the Kerchunk reference file for each URL from the list of URLs in parallel\n\n%%time\n\nreference_files = []\nfor url in urls:\n ref = dask.delayed(gen_json)(url)\n reference_files.append(ref)\n\nreference_files_compute = dask.compute(*reference_files)\n\nCPU times: user 29 s, sys: 11.1 s, total: 40 s\nWall time: 11min 6s\n\n\nCreate a python list with the paths to the reference files.\n\nreference_list = sorted(glob('./jsons/*.json'))\nreference_list\n\n['./jsons/MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190502.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190503.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190504.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190505.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190506.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190507.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190508.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190509.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190510.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190511.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190512.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190513.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190514.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190515.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190516.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190517.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190518.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190519.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190520.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190521.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190522.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190523.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190524.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190525.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190526.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190527.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190528.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190529.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190530.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190531.nc4.json']" + }, + { + "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#read-single-netcdf4-using-kerchunk-reference-file", + "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#read-single-netcdf4-using-kerchunk-reference-file", + "title": "GES DISC - MERRA2", + "section": "Read single netCDF4 using Kerchunk reference file", + "text": "Read single netCDF4 using Kerchunk reference file\nOpen the first reference file to read into an xarray dataset\n\nwith open(reference_list[0]) as j:\n reference = ujson.load(j)\n\nSet configurations options\n\ns_opts = {'skip_instance_cache':True} #json\nr_opts = {'anon':False, \n 'key':temp_creds_req['accessKeyId'], \n 'secret':temp_creds_req['secretAccessKey'], \n 'token':temp_creds_req['sessionToken']} #ncfiles\n\n\nfs_single = fsspec.filesystem(\"reference\",\n fo=reference,\n ref_storage_args=s_opts,\n remote_protocol='s3', \n remote_options=r_opts)\n\nRead in a single reference object. We get a lot of SerializationWarnings which are ignored here using the warning package.\nNOTE, the fill value, data range, min value, and max value DO NOT match the source file. Will need to look into this more.\n\n%%time\n\nm = fs_single.get_mapper(\"\")\nds_single = xr.open_dataset(m, engine=\"zarr\", backend_kwargs={'consolidated':False}, chunks={})\nds_single\n\nCPU times: user 142 ms, sys: 3.29 ms, total: 146 ms\nWall time: 354 ms\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (time: 24, lat: 361, lon: 576)\nCoordinates:\n * lat (lat) float64 -90.0 -89.5 -89.0 -88.5 ... 88.5 89.0 89.5 90.0\n * lon (lon) float64 -180.0 -179.4 -178.8 -178.1 ... 178.1 178.8 179.4\n * time (time) datetime64[ns] 2019-05-01T00:30:00 ... 2019-05-01T23:30:00\nData variables: (12/47)\n CLDPRS (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n CLDTMP (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n DISPH (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n H1000 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n H250 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n H500 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n ... ...\n V250 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n V2M (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n V500 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n V50M (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n V850 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n ZLCL (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\nAttributes: (12/30)\n Comment: GMAO filename: d5124_m2_jan10.tavg1_2d...\n Contact: http://gmao.gsfc.nasa.gov\n Conventions: CF-1\n DataResolution: 0.5 x 0.625\n EasternmostLongitude: 179.375\n Filename: MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4\n ... ...\n TemporalRange: 1980-01-01 -> 2016-12-31\n Title: MERRA2 tavg1_2d_slv_Nx: 2d,1-Hourly,Ti...\n VersionID: 5.12.4\n WesternmostLongitude: -180.0\n identifier_product_doi: 10.5067/VJAFPLI1CSIV\n identifier_product_doi_authority: http://dx.doi.org/xarray.DatasetDimensions:time: 24lat: 361lon: 576Coordinates: (3)lat(lat)float64-90.0 -89.5 -89.0 ... 89.5 90.0long_name :latitudeunits :degrees_northvalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array([-90. , -89.5, -89. , ..., 89. , 89.5, 90. ])lon(lon)float64-180.0 -179.4 ... 178.8 179.4long_name :longitudeunits :degrees_eastvalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array([-180. , -179.375, -178.75 , ..., 178.125, 178.75 , 179.375])time(time)datetime64[ns]2019-05-01T00:30:00 ... 2019-05-...begin_date :20190501begin_time :3000long_name :timetime_increment :10000valid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array(['2019-05-01T00:30:00.000000000', '2019-05-01T01:30:00.000000000',\n '2019-05-01T02:30:00.000000000', '2019-05-01T03:30:00.000000000',\n '2019-05-01T04:30:00.000000000', '2019-05-01T05:30:00.000000000',\n '2019-05-01T06:30:00.000000000', '2019-05-01T07:30:00.000000000',\n '2019-05-01T08:30:00.000000000', '2019-05-01T09:30:00.000000000',\n '2019-05-01T10:30:00.000000000', '2019-05-01T11:30:00.000000000',\n '2019-05-01T12:30:00.000000000', '2019-05-01T13:30:00.000000000',\n '2019-05-01T14:30:00.000000000', '2019-05-01T15:30:00.000000000',\n '2019-05-01T16:30:00.000000000', '2019-05-01T17:30:00.000000000',\n '2019-05-01T18:30:00.000000000', '2019-05-01T19:30:00.000000000',\n '2019-05-01T20:30:00.000000000', '2019-05-01T21:30:00.000000000',\n '2019-05-01T22:30:00.000000000', '2019-05-01T23:30:00.000000000'],\n dtype='datetime64[ns]')Data variables: (47)CLDPRS(time, lat, lon)float32dask.array<chunksize=(1, 91, 144), meta=np.ndarray>fmissing_value :999999986991104.0long_name :cloud_top_pressurestandard_name :cloud_top_pressureunits :Pavalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nCLDTMP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ncloud_top_temperature\n\nstandard_name :\n\ncloud_top_temperature\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nDISPH\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nzero_plane_displacement_height\n\nstandard_name :\n\nzero_plane_displacement_height\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH1000\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nheight_at_1000_mb\n\nstandard_name :\n\nheight_at_1000_mb\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nheight_at_250_hPa\n\nstandard_name :\n\nheight_at_250_hPa\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nheight_at_500_hPa\n\nstandard_name :\n\nheight_at_500_hPa\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nheight_at_850_hPa\n\nstandard_name :\n\nheight_at_850_hPa\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nOMEGA500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nomega_at_500_hPa\n\nstandard_name :\n\nomega_at_500_hPa\n\nunits :\n\nPa s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nPBLTOP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\npbltop_pressure\n\nstandard_name :\n\npbltop_pressure\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nPS\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nsurface_pressure\n\nstandard_name :\n\nsurface_pressure\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nspecific_humidity_at_250_hPa\n\nstandard_name :\n\nspecific_humidity_at_250_hPa\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nspecific_humidity_at_500_hPa\n\nstandard_name :\n\nspecific_humidity_at_500_hPa\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nspecific_humidity_at_850_hPa\n\nstandard_name :\n\nspecific_humidity_at_850_hPa\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQV10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n10-meter_specific_humidity\n\nstandard_name :\n\n10-meter_specific_humidity\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQV2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n2-meter_specific_humidity\n\nstandard_name :\n\n2-meter_specific_humidity\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSLP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nsea_level_pressure\n\nstandard_name :\n\nsea_level_pressure\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n10-meter_air_temperature\n\nstandard_name :\n\n10-meter_air_temperature\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nair_temperature_at_250_hPa\n\nstandard_name :\n\nair_temperature_at_250_hPa\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n2-meter_air_temperature\n\nstandard_name :\n\n2-meter_air_temperature\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2MDEW\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ndew_point_temperature_at_2_m\n\nstandard_name :\n\ndew_point_temperature_at_2_m\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2MWET\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nwet_bulb_temperature_at_2_m\n\nstandard_name :\n\nwet_bulb_temperature_at_2_m\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nair_temperature_at_500_hPa\n\nstandard_name :\n\nair_temperature_at_500_hPa\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nair_temperature_at_850_hPa\n\nstandard_name :\n\nair_temperature_at_850_hPa\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTO3\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_column_ozone\n\nstandard_name :\n\ntotal_column_ozone\n\nunits :\n\nDobsons\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTOX\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_column_odd_oxygen\n\nstandard_name :\n\ntotal_column_odd_oxygen\n\nunits :\n\nkg m-2\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQI\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_precipitable_ice_water\n\nstandard_name :\n\ntotal_precipitable_ice_water\n\nunits :\n\nkg m-2\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQL\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_precipitable_liquid_water\n\nstandard_name :\n\ntotal_precipitable_liquid_water\n\nunits :\n\nkg m-2\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQV\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_precipitable_water_vapor\n\nstandard_name :\n\ntotal_precipitable_water_vapor\n\nunits :\n\nkg m-2\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPB\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_pressure_based_on_blended_estimate\n\nstandard_name :\n\ntropopause_pressure_based_on_blended_estimate\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPT\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_pressure_based_on_thermal_estimate\n\nstandard_name :\n\ntropopause_pressure_based_on_thermal_estimate\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPV\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_pressure_based_on_EPV_estimate\n\nstandard_name :\n\ntropopause_pressure_based_on_EPV_estimate\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPQ\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_specific_humidity_using_blended_TROPP_estimate\n\nstandard_name :\n\ntropopause_specific_humidity_using_blended_TROPP_estimate\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPT\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_temperature_using_blended_TROPP_estimate\n\nstandard_name :\n\ntropopause_temperature_using_blended_TROPP_estimate\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTS\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nsurface_skin_temperature\n\nstandard_name :\n\nsurface_skin_temperature\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n10-meter_eastward_wind\n\nstandard_name :\n\n10-meter_eastward_wind\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\neastward_wind_at_250_hPa\n\nstandard_name :\n\neastward_wind_at_250_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n2-meter_eastward_wind\n\nstandard_name :\n\n2-meter_eastward_wind\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\neastward_wind_at_500_hPa\n\nstandard_name :\n\neastward_wind_at_500_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU50M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\neastward_wind_at_50_meters\n\nstandard_name :\n\neastward_wind_at_50_meters\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\neastward_wind_at_850_hPa\n\nstandard_name :\n\neastward_wind_at_850_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n10-meter_northward_wind\n\nstandard_name :\n\n10-meter_northward_wind\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nnorthward_wind_at_250_hPa\n\nstandard_name :\n\nnorthward_wind_at_250_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n2-meter_northward_wind\n\nstandard_name :\n\n2-meter_northward_wind\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nnorthward_wind_at_500_hPa\n\nstandard_name :\n\nnorthward_wind_at_500_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV50M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nnorthward_wind_at_50_meters\n\nstandard_name :\n\nnorthward_wind_at_50_meters\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nnorthward_wind_at_850_hPa\n\nstandard_name :\n\nnorthward_wind_at_850_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nZLCL\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nlifting_condensation_level\n\nstandard_name :\n\nlifting_condensation_level\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nAttributes: (30)Comment :GMAO filename: d5124_m2_jan10.tavg1_2d_slv_Nx.20190501.nc4Contact :http://gmao.gsfc.nasa.govConventions :CF-1DataResolution :0.5 x 0.625EasternmostLongitude :179.375Filename :MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4Format :NetCDF-4/HDF-5GranuleID :MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4History :Original file generated: Sat May 11 22:08:52 2019 GMTInstitution :NASA Global Modeling and Assimilation OfficeLatitudeResolution :0.5LongName :MERRA2 tavg1_2d_slv_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Single-Level DiagnosticsLongitudeResolution :0.625NorthernmostLatitude :90.0ProductionDateTime :Original file generated: Sat May 11 22:08:52 2019 GMTRangeBeginningDate :2019-05-01RangeBeginningTime :00:00:00.000000RangeEndingDate :2019-05-01RangeEndingTime :23:59:59.000000References :http://gmao.gsfc.nasa.govShortName :M2T1NXSLVSource :CVS tag: GEOSadas-5_12_4_p16_sp3_M2-OPSSouthernmostLatitude :-90.0SpatialCoverage :globalTemporalRange :1980-01-01 -> 2016-12-31Title :MERRA2 tavg1_2d_slv_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Single-Level DiagnosticsVersionID :5.12.4WesternmostLongitude :-180.0identifier_product_doi :10.5067/VJAFPLI1CSIVidentifier_product_doi_authority :http://dx.doi.org/" + }, + { + "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#read-multiple-netcdf4-files-using-kerchunk-reference-file", + "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#read-multiple-netcdf4-files-using-kerchunk-reference-file", + "title": "GES DISC - MERRA2", + "section": "Read multiple netCDF4 files using Kerchunk reference file", + "text": "Read multiple netCDF4 files using Kerchunk reference file\nCombine the individual reference files into a single time series reference object\n\n%%time\n\nds_k =[]\nfor ref in reference_list:\n s_opts = s_opts\n r_opts = r_opts\n fs = fsspec.filesystem(\"reference\",\n fo=ref,\n ref_storage_args=s_opts,\n remote_protocol='s3',\n remote_options=r_opts)\n m = fs.get_mapper(\"\")\n ds_k.append(xr.open_dataset(m, engine=\"zarr\", backend_kwargs={'consolidated':False}, chunks={}))\n \nds_multi = xr.concat(ds_k, dim='time')\n \nds_multi\n\nCPU times: user 8.93 s, sys: 174 ms, total: 9.1 s\nWall time: 14.9 s\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (time: 744, lat: 361, lon: 576)\nCoordinates:\n * lat (lat) float64 -90.0 -89.5 -89.0 -88.5 ... 88.5 89.0 89.5 90.0\n * lon (lon) float64 -180.0 -179.4 -178.8 -178.1 ... 178.1 178.8 179.4\n * time (time) datetime64[ns] 2019-05-01T00:30:00 ... 2019-05-31T23:30:00\nData variables: (12/47)\n CLDPRS (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n CLDTMP (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n DISPH (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n H1000 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n H250 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n H500 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n ... ...\n V250 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n V2M (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n V500 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n V50M (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n V850 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n ZLCL (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\nAttributes: (12/30)\n Comment: GMAO filename: d5124_m2_jan10.tavg1_2d...\n Contact: http://gmao.gsfc.nasa.gov\n Conventions: CF-1\n DataResolution: 0.5 x 0.625\n EasternmostLongitude: 179.375\n Filename: MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4\n ... ...\n TemporalRange: 1980-01-01 -> 2016-12-31\n Title: MERRA2 tavg1_2d_slv_Nx: 2d,1-Hourly,Ti...\n VersionID: 5.12.4\n WesternmostLongitude: -180.0\n identifier_product_doi: 10.5067/VJAFPLI1CSIV\n identifier_product_doi_authority: http://dx.doi.org/xarray.DatasetDimensions:time: 744lat: 361lon: 576Coordinates: (3)lat(lat)float64-90.0 -89.5 -89.0 ... 89.5 90.0long_name :latitudeunits :degrees_northvalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array([-90. , -89.5, -89. , ..., 89. , 89.5, 90. ])lon(lon)float64-180.0 -179.4 ... 178.8 179.4long_name :longitudeunits :degrees_eastvalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array([-180. , -179.375, -178.75 , ..., 178.125, 178.75 , 179.375])time(time)datetime64[ns]2019-05-01T00:30:00 ... 2019-05-...begin_date :20190501begin_time :3000long_name :timetime_increment :10000valid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array(['2019-05-01T00:30:00.000000000', '2019-05-01T01:30:00.000000000',\n '2019-05-01T02:30:00.000000000', ..., '2019-05-31T21:30:00.000000000',\n '2019-05-31T22:30:00.000000000', '2019-05-31T23:30:00.000000000'],\n dtype='datetime64[ns]')Data variables: (47)CLDPRS(time, lat, lon)float32dask.array<chunksize=(1, 91, 144), meta=np.ndarray>fmissing_value :999999986991104.0long_name :cloud_top_pressurestandard_name :cloud_top_pressureunits :Pavalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nCLDTMP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ncloud_top_temperature\n\nstandard_name :\n\ncloud_top_temperature\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nDISPH\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nzero_plane_displacement_height\n\nstandard_name :\n\nzero_plane_displacement_height\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH1000\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nheight_at_1000_mb\n\nstandard_name :\n\nheight_at_1000_mb\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nheight_at_250_hPa\n\nstandard_name :\n\nheight_at_250_hPa\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nheight_at_500_hPa\n\nstandard_name :\n\nheight_at_500_hPa\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nheight_at_850_hPa\n\nstandard_name :\n\nheight_at_850_hPa\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nOMEGA500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nomega_at_500_hPa\n\nstandard_name :\n\nomega_at_500_hPa\n\nunits :\n\nPa s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nPBLTOP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\npbltop_pressure\n\nstandard_name :\n\npbltop_pressure\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nPS\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nsurface_pressure\n\nstandard_name :\n\nsurface_pressure\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nspecific_humidity_at_250_hPa\n\nstandard_name :\n\nspecific_humidity_at_250_hPa\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nspecific_humidity_at_500_hPa\n\nstandard_name :\n\nspecific_humidity_at_500_hPa\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nspecific_humidity_at_850_hPa\n\nstandard_name :\n\nspecific_humidity_at_850_hPa\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQV10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n10-meter_specific_humidity\n\nstandard_name :\n\n10-meter_specific_humidity\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQV2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n2-meter_specific_humidity\n\nstandard_name :\n\n2-meter_specific_humidity\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSLP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nsea_level_pressure\n\nstandard_name :\n\nsea_level_pressure\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n10-meter_air_temperature\n\nstandard_name :\n\n10-meter_air_temperature\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nair_temperature_at_250_hPa\n\nstandard_name :\n\nair_temperature_at_250_hPa\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n2-meter_air_temperature\n\nstandard_name :\n\n2-meter_air_temperature\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2MDEW\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ndew_point_temperature_at_2_m\n\nstandard_name :\n\ndew_point_temperature_at_2_m\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2MWET\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nwet_bulb_temperature_at_2_m\n\nstandard_name :\n\nwet_bulb_temperature_at_2_m\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nair_temperature_at_500_hPa\n\nstandard_name :\n\nair_temperature_at_500_hPa\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nair_temperature_at_850_hPa\n\nstandard_name :\n\nair_temperature_at_850_hPa\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTO3\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_column_ozone\n\nstandard_name :\n\ntotal_column_ozone\n\nunits :\n\nDobsons\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTOX\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_column_odd_oxygen\n\nstandard_name :\n\ntotal_column_odd_oxygen\n\nunits :\n\nkg m-2\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQI\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_precipitable_ice_water\n\nstandard_name :\n\ntotal_precipitable_ice_water\n\nunits :\n\nkg m-2\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQL\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_precipitable_liquid_water\n\nstandard_name :\n\ntotal_precipitable_liquid_water\n\nunits :\n\nkg m-2\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQV\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_precipitable_water_vapor\n\nstandard_name :\n\ntotal_precipitable_water_vapor\n\nunits :\n\nkg m-2\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPB\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_pressure_based_on_blended_estimate\n\nstandard_name :\n\ntropopause_pressure_based_on_blended_estimate\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPT\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_pressure_based_on_thermal_estimate\n\nstandard_name :\n\ntropopause_pressure_based_on_thermal_estimate\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPV\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_pressure_based_on_EPV_estimate\n\nstandard_name :\n\ntropopause_pressure_based_on_EPV_estimate\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPQ\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_specific_humidity_using_blended_TROPP_estimate\n\nstandard_name :\n\ntropopause_specific_humidity_using_blended_TROPP_estimate\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPT\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_temperature_using_blended_TROPP_estimate\n\nstandard_name :\n\ntropopause_temperature_using_blended_TROPP_estimate\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTS\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nsurface_skin_temperature\n\nstandard_name :\n\nsurface_skin_temperature\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n10-meter_eastward_wind\n\nstandard_name :\n\n10-meter_eastward_wind\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\neastward_wind_at_250_hPa\n\nstandard_name :\n\neastward_wind_at_250_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n2-meter_eastward_wind\n\nstandard_name :\n\n2-meter_eastward_wind\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\neastward_wind_at_500_hPa\n\nstandard_name :\n\neastward_wind_at_500_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU50M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\neastward_wind_at_50_meters\n\nstandard_name :\n\neastward_wind_at_50_meters\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\neastward_wind_at_850_hPa\n\nstandard_name :\n\neastward_wind_at_850_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n10-meter_northward_wind\n\nstandard_name :\n\n10-meter_northward_wind\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nnorthward_wind_at_250_hPa\n\nstandard_name :\n\nnorthward_wind_at_250_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n2-meter_northward_wind\n\nstandard_name :\n\n2-meter_northward_wind\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nnorthward_wind_at_500_hPa\n\nstandard_name :\n\nnorthward_wind_at_500_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV50M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nnorthward_wind_at_50_meters\n\nstandard_name :\n\nnorthward_wind_at_50_meters\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nnorthward_wind_at_850_hPa\n\nstandard_name :\n\nnorthward_wind_at_850_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nZLCL\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nlifting_condensation_level\n\nstandard_name :\n\nlifting_condensation_level\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nAttributes: (30)Comment :GMAO filename: d5124_m2_jan10.tavg1_2d_slv_Nx.20190501.nc4Contact :http://gmao.gsfc.nasa.govConventions :CF-1DataResolution :0.5 x 0.625EasternmostLongitude :179.375Filename :MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4Format :NetCDF-4/HDF-5GranuleID :MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4History :Original file generated: Sat May 11 22:08:52 2019 GMTInstitution :NASA Global Modeling and Assimilation OfficeLatitudeResolution :0.5LongName :MERRA2 tavg1_2d_slv_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Single-Level DiagnosticsLongitudeResolution :0.625NorthernmostLatitude :90.0ProductionDateTime :Original file generated: Sat May 11 22:08:52 2019 GMTRangeBeginningDate :2019-05-01RangeBeginningTime :00:00:00.000000RangeEndingDate :2019-05-01RangeEndingTime :23:59:59.000000References :http://gmao.gsfc.nasa.govShortName :M2T1NXSLVSource :CVS tag: GEOSadas-5_12_4_p16_sp3_M2-OPSSouthernmostLatitude :-90.0SpatialCoverage :globalTemporalRange :1980-01-01 -> 2016-12-31Title :MERRA2 tavg1_2d_slv_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Single-Level DiagnosticsVersionID :5.12.4WesternmostLongitude :-180.0identifier_product_doi :10.5067/VJAFPLI1CSIVidentifier_product_doi_authority :http://dx.doi.org/\n\n\nAgains, the fill value, data range, min value, and max value DO NOT match the source file. TODO: explore why the values are different\n\nds_multi['T500']\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray 'T500' (time: 744, lat: 361, lon: 576)>\ndask.array<concatenate, shape=(744, 361, 576), dtype=float32, chunksize=(1, 91, 144), chunktype=numpy.ndarray>\nCoordinates:\n * lat (lat) float64 -90.0 -89.5 -89.0 -88.5 -88.0 ... 88.5 89.0 89.5 90.0\n * lon (lon) float64 -180.0 -179.4 -178.8 -178.1 ... 178.1 178.8 179.4\n * time (time) datetime64[ns] 2019-05-01T00:30:00 ... 2019-05-31T23:30:00\nAttributes:\n fmissing_value: 999999986991104.0\n long_name: air_temperature_at_500_hPa\n standard_name: air_temperature_at_500_hPa\n units: K\n valid_range: [-999999986991104.0, 999999986991104.0]\n vmax: 999999986991104.0\n vmin: -999999986991104.0xarray.DataArray'T500'time: 744lat: 361lon: 576dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\nCoordinates: (3)lat(lat)float64-90.0 -89.5 -89.0 ... 89.5 90.0long_name :latitudeunits :degrees_northvalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array([-90. , -89.5, -89. , ..., 89. , 89.5, 90. ])lon(lon)float64-180.0 -179.4 ... 178.8 179.4long_name :longitudeunits :degrees_eastvalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array([-180. , -179.375, -178.75 , ..., 178.125, 178.75 , 179.375])time(time)datetime64[ns]2019-05-01T00:30:00 ... 2019-05-...begin_date :20190501begin_time :3000long_name :timetime_increment :10000valid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array(['2019-05-01T00:30:00.000000000', '2019-05-01T01:30:00.000000000',\n '2019-05-01T02:30:00.000000000', ..., '2019-05-31T21:30:00.000000000',\n '2019-05-31T22:30:00.000000000', '2019-05-31T23:30:00.000000000'],\n dtype='datetime64[ns]')Attributes: (7)fmissing_value :999999986991104.0long_name :air_temperature_at_500_hPastandard_name :air_temperature_at_500_hPaunits :Kvalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0\n\n\n\n# Commenting for quarto site render\n# ds_multi['T500'].hvplot.image(x='lon', y='lat')" + }, + { + "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#references", + "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#references", + "title": "GES DISC - MERRA2", + "section": "References", + "text": "References\n\nhttps://github.com/fsspec/kerchunk\nhttps://medium.com/pangeo/fake-it-until-you-make-it-reading-goes-netcdf4-data-on-aws-s3-as-zarr-for-rapid-data-access-61e33f8fe685\nhttps://medium.com/pangeo/cloud-performant-reading-of-netcdf4-hdf5-data-using-the-zarr-library-1a95c5c92314" + }, + { + "objectID": "examples/LPDAAC/Find_and_Access_HLS_PointBuffer.html", + "href": "examples/LPDAAC/Find_and_Access_HLS_PointBuffer.html", + "title": "Create buffer around point location", "section": "", - "text": "All documentation and chapter materials in this repository are made available under the Creative Commons Attribution license. The following is a human-readable summary of (and not a substitute for) the full legal text of the CC BY 4.0 license.\nYou are free:\n\nto Share—copy and redistribute the material in any medium or format\nto Adapt—remix, transform, and build upon the material\n\nfor any purpose, even commercially.\nThe licensor cannot revoke these freedoms as long as you follow the license terms.\nUnder the following terms:\n\nAttribution—You must give appropriate credit (mentioning that your work is derived from work that is Copyright © Openscapes and, where practical, linking to https://github.com/openscapes/approach-guide), provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.\nNo additional restrictions—You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.\n\nWith the understanding that:\n\nYou do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation.\nNo warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material." + "text": "import geopandas\nimport json\nimport shapely\nimport shapely.geometry\nfrom shapely.ops import transform\nfrom shapely.geometry import Point\nfrom shapely.geometry import Polygon\nimport geoviews as gv\nfrom pystac_client import Client \nimport xarray\nimport rasterio as rio\nimport rioxarray\nimport os\ngv.extension('bokeh', 'matplotlib')\nlon = -84.0106\nlat = 10.432\nbuffer = 0.001 # decimal degrees equal to ~222 meters\nloc = Point(lon, lat).buffer(buffer)\ntype(loc)\n\nshapely.geometry.polygon.Polygon\nbase = gv.tile_sources.EsriImagery.opts(width=650, height=500)\nsample_loc = gv.Polygons(loc).opts(line_color='yellow', line_width=10, color=None)\nbase * sample_loc\nTranslate the shapely polygon to a GeoJSON Polygon Object, which is what the STAC API expects.\nroi = json.loads(geopandas.GeoSeries([loc]).to_json())\nroi\n\n{'type': 'FeatureCollection',\n 'features': [{'id': '0',\n 'type': 'Feature',\n 'properties': {},\n 'geometry': {'type': 'Polygon',\n 'coordinates': [[[-84.00959999999999, 10.432],\n [-84.00960481527332, 10.43190198285967],\n [-84.00961921471959, 10.431804909677984],\n [-84.00964305966427, 10.431709715322746],\n [-84.00967612046749, 10.431617316567635],\n [-84.00971807873565, 10.431528603263175],\n [-84.00976853038769, 10.43144442976698],\n [-84.00982698954664, 10.431365606715836],\n [-84.0098928932188, 10.431292893218814],\n [-84.00996560671584, 10.431226989546637],\n [-84.01004442976698, 10.431168530387698],\n [-84.01012860326317, 10.431118078735652],\n [-84.01021731656763, 10.43107612046749],\n [-84.01030971532275, 10.431043059664269],\n [-84.01040490967799, 10.431019214719598],\n [-84.01050198285967, 10.431004815273328],\n [-84.0106, 10.431000000000001],\n [-84.01069801714033, 10.431004815273328],\n [-84.010795090322, 10.431019214719598],\n [-84.01089028467725, 10.431043059664269],\n [-84.01098268343236, 10.43107612046749],\n [-84.01107139673682, 10.431118078735652],\n [-84.01115557023302, 10.431168530387698],\n [-84.01123439328416, 10.431226989546637],\n [-84.01130710678119, 10.431292893218814],\n [-84.01137301045335, 10.431365606715836],\n [-84.0114314696123, 10.43144442976698],\n [-84.01148192126435, 10.431528603263175],\n [-84.0115238795325, 10.431617316567635],\n [-84.01155694033572, 10.431709715322746],\n [-84.0115807852804, 10.431804909677984],\n [-84.01159518472667, 10.43190198285967],\n [-84.0116, 10.432],\n [-84.01159518472667, 10.43209801714033],\n [-84.0115807852804, 10.432195090322017],\n [-84.01155694033572, 10.432290284677254],\n [-84.0115238795325, 10.432382683432365],\n [-84.01148192126435, 10.432471396736826],\n [-84.0114314696123, 10.43255557023302],\n [-84.01137301045335, 10.432634393284165],\n [-84.01130710678119, 10.432707106781187],\n [-84.01123439328416, 10.432773010453364],\n [-84.01115557023302, 10.432831469612303],\n [-84.01107139673682, 10.432881921264348],\n [-84.01098268343236, 10.432923879532511],\n [-84.01089028467725, 10.432956940335732],\n [-84.010795090322, 10.432980785280403],\n [-84.01069801714033, 10.432995184726673],\n [-84.0106, 10.433],\n [-84.01050198285967, 10.432995184726673],\n [-84.01040490967799, 10.432980785280403],\n [-84.01030971532275, 10.432956940335732],\n [-84.01021731656763, 10.432923879532511],\n [-84.01012860326317, 10.432881921264348],\n [-84.01004442976698, 10.432831469612303],\n [-84.00996560671584, 10.432773010453364],\n [-84.0098928932188, 10.432707106781187],\n [-84.00982698954664, 10.432634393284165],\n [-84.00976853038769, 10.43255557023302],\n [-84.00971807873565, 10.432471396736826],\n [-84.00967612046749, 10.432382683432365],\n [-84.00964305966427, 10.432290284677254],\n [-84.00961921471959, 10.432195090322017],\n [-84.00960481527332, 10.43209801714033],\n [-84.00959999999999, 10.432]]]},\n 'bbox': [-84.0116, 10.431000000000001, -84.00959999999999, 10.433]}],\n 'bbox': [-84.0116, 10.431000000000001, -84.00959999999999, 10.433]}\nGeoJSON Polygon Object vertices are expected to be in counterclockwise order. The current object vertices are in clockwise order. Below we reverse the order of the vertices and assign them back to our roi object.\nroi['features'][0]['geometry']['coordinates'][0] = roi['features'][0]['geometry']['coordinates'][0][::-1]" }, { - "objectID": "LICENSE.html#software", - "href": "LICENSE.html#software", - "title": "LICENSE", + "objectID": "examples/LPDAAC/Find_and_Access_HLS_PointBuffer.html#find-data-using-the-stac-api", + "href": "examples/LPDAAC/Find_and_Access_HLS_PointBuffer.html#find-data-using-the-stac-api", + "title": "Create buffer around point location", + "section": "Find data using the STAC API", + "text": "Find data using the STAC API\nSpecify the STAC API endpoint\n\nSTAC_URL = 'https://cmr.earthdata.nasa.gov/stac'\n\nConnect to the LPCLOUD catalog using PySTAC Client\n\ncatalog = Client.open(f'{STAC_URL}/LPCLOUD/')\n\nSubmit a query for: - HLSL30 and HLSS30 - The roi’s geometry - Dates between 05-01-2021 and 06-30-2022\n\nsearch = catalog.search(\n collections = ['HLSL30.v2.0', 'HLSS30.v2.0'],\n intersects = roi['features'][0]['geometry'],\n datetime = '2021-05/2022-06'\n) \n\n\nsearch.matched()\n\n87\n\n\nGet all the STAC Items/granules\n\nitems = search.get_all_items()\n\n\nlen(items)\n\n87\n\n\nGet the first STAC Item from the list and print off the available assets\n\nitems[0].assets\n\n{'B07': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B07.tif>,\n 'B01': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B01.tif>,\n 'B04': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B04.tif>,\n 'B06': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B06.tif>,\n 'VZA': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.VZA.tif>,\n 'Fmask': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.Fmask.tif>,\n 'B02': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B02.tif>,\n 'SAA': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.SAA.tif>,\n 'B11': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B11.tif>,\n 'B03': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B03.tif>,\n 'VAA': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.VAA.tif>,\n 'B10': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B10.tif>,\n 'B05': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B05.tif>,\n 'B09': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B09.tif>,\n 'SZA': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.SZA.tif>,\n 'browse': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.jpg>,\n 'metadata': <Asset href=https://cmr.earthdata.nasa.gov/search/concepts/G2144745416-LPCLOUD.xml>}\n\n\nGrab the URL for the B04 asset\n\ndata_url = items[0].assets['B04'].href\ndata_url\n\n'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B04.tif'\n\n\nNOTE: The above returns the HTTPS URL. For direct S3 access of cloud assets the S3 URL is need. See the following resource on how to generate the S3 URL: https://nasa-openscapes.github.io/2021-Cloud-Hackathon/tutorials/02_Data_Discovery_CMR-STAC_API.html" + }, + { + "objectID": "examples/LPDAAC/Find_and_Access_HLS_PointBuffer.html#read-in-the-url-using-rioxarray", + "href": "examples/LPDAAC/Find_and_Access_HLS_PointBuffer.html#read-in-the-url-using-rioxarray", + "title": "Create buffer around point location", + "section": "Read in the URL using rioxarray", + "text": "Read in the URL using rioxarray\nNOTE: The example below show how to access the asset via HTTPS. An example for performing direct S3 access can be found here: https://nasa-openscapes.github.io/2021-Cloud-Hackathon/tutorials/05_Data_Access_Direct_S3.html\nFirst we need to set our gdal configs. Make sure a .netrc file has been created in your home directory\n\nrio_env = rio.Env(GDAL_DISABLE_READDIR_ON_OPEN='EMPTY',\n GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/cookies.txt'),\n GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/cookies.txt'))\nrio_env.__enter__()\n\n<rasterio.env.Env at 0x7f846be57f70>\n\n\n\nda = rioxarray.open_rasterio(data_url)\n\n\nda\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (band: 1, y: 3660, x: 3660)>\n[13395600 values with dtype=int16]\nCoordinates:\n * band (band) int64 1\n * x (x) float64 8e+05 8e+05 8.001e+05 ... 9.097e+05 9.098e+05\n * y (y) float64 1.2e+06 1.2e+06 1.2e+06 ... 1.09e+06 1.09e+06\n spatial_ref int64 0\nAttributes:\n _FillValue: -9999.0\n scale_factor: 0.0001\n add_offset: 0.0\n long_name: Redxarray.DataArrayband: 1y: 3660x: 3660...[13395600 values with dtype=int16]Coordinates: (4)band(band)int641array([1])x(x)float648e+05 8e+05 ... 9.097e+05 9.098e+05array([799995., 800025., 800055., ..., 909705., 909735., 909765.])y(y)float641.2e+06 1.2e+06 ... 1.09e+06array([1199985., 1199955., 1199925., ..., 1090275., 1090245., 1090215.])spatial_ref()int640crs_wkt :PROJCS[\"UTM Zone 16, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not_specified_based_on_WGS_84_spheroid\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-87],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not_specified_based_on_WGS_84_spheroidprojected_crs_name :UTM Zone 16, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-87.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 16, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not_specified_based_on_WGS_84_spheroid\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-87],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :799980.0 30.0 0.0 1200000.0 0.0 -30.0array(0)Attributes: (4)_FillValue :-9999.0scale_factor :0.0001add_offset :0.0long_name :Red\n\n\nDrop the ‘band’ coordinate\n\nda.squeeze('band', drop=True)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (y: 3660, x: 3660)>\n[13395600 values with dtype=int16]\nCoordinates:\n * x (x) float64 8e+05 8e+05 8.001e+05 ... 9.097e+05 9.098e+05\n * y (y) float64 1.2e+06 1.2e+06 1.2e+06 ... 1.09e+06 1.09e+06\n spatial_ref int64 0\nAttributes:\n _FillValue: -9999.0\n scale_factor: 0.0001\n add_offset: 0.0\n long_name: Redxarray.DataArrayy: 3660x: 3660...[13395600 values with dtype=int16]Coordinates: (3)x(x)float648e+05 8e+05 ... 9.097e+05 9.098e+05array([799995., 800025., 800055., ..., 909705., 909735., 909765.])y(y)float641.2e+06 1.2e+06 ... 1.09e+06array([1199985., 1199955., 1199925., ..., 1090275., 1090245., 1090215.])spatial_ref()int640crs_wkt :PROJCS[\"UTM Zone 16, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not_specified_based_on_WGS_84_spheroid\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-87],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not_specified_based_on_WGS_84_spheroidprojected_crs_name :UTM Zone 16, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-87.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 16, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not_specified_based_on_WGS_84_spheroid\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-87],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :799980.0 30.0 0.0 1200000.0 0.0 -30.0array(0)Attributes: (4)_FillValue :-9999.0scale_factor :0.0001add_offset :0.0long_name :Red" + }, + { + "objectID": "examples/USGS_Landsat/Landsat_C2_Search_Access.html", + "href": "examples/USGS_Landsat/Landsat_C2_Search_Access.html", + "title": "USGS Landsat Cloud Data - Search and Access", "section": "", - "text": "Except where otherwise noted, the software and infrastructure provided by the The Turing Way Community are made available under the OSI-approved MIT license.\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\nTHE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE." + "text": "import rasterio as rio\nfrom rasterio.session import AWSSession\nimport rioxarray\nimport boto3\nimport json\nfrom pystac_client import Client\nimport hvplot.xarray\nSeach for LS Scenes using STAC API\nstac_url = 'https://landsatlook.usgs.gov/stac-server'\nls_cat = Client.open(stac_url)\nls_cat\n\n<Client id=stac-server>\n[f'{x.title} --- {x.id}' for x in ls_cat.get_children()]\n\n['Landsat Collection 1 Level-2 Albers Surface Temperature (ST) Product --- landsat-c1l2alb-st',\n 'Landsat Collection 2 Level-2 UTM Surface Reflectance (SR) Product --- landsat-c2l2-sr',\n 'Landsat Collection 2 Level-2 UTM Surface Temperature (ST) Product --- landsat-c2l2-st',\n 'Landsat Collection 2 Analysis Ready Data (ARD) Level-2 UTM Surface Temperature (ST) Product --- landsat-c2ard-st',\n 'Landsat Collection 2 Level-2 Albers Top of Atmosphere Brightness Temperature (BT) Product --- landsat-c2l2alb-bt',\n 'Landsat Collection 2 Analysis Ready Data (ARD) Level-2 UTM Top of Atmosphere Brightness Temperature (BT) Product --- landsat-c2ard-bt',\n 'Landsat Collection 2 Level-3 Fractional Snow Covered Area (fSCA) Product --- landsat-c2l3-fsca',\n 'Landsat Collection 1 Level-2 Albers Top of Atmosphere Brightness Temperature (BT) Product --- landsat-c1l2alb-bt',\n 'Landsat Collection 2 Level-1 Product --- landsat-c2l1',\n 'Landsat Collection 2 Level-3 Burned Area (BA) Product --- landsat-c2l3-ba',\n 'Landsat Collection 1 Level-2 Albers Top of Atmosphere (TA) Reflectance Product --- landsat-c1l2alb-ta',\n 'Landsat Collection 2 Level-2 Albers Surface Temperature (ST) Product --- landsat-c2l2alb-st',\n 'Landsat Collection 2 Analysis Ready Data (ARD) Level-2 UTM Surface Reflectance (SR) Product --- landsat-c2ard-sr',\n 'Landsat Collection 1 Level-1 Product --- landsat-c1l1',\n 'Landsat Collection 1 Level-2 Albers Surface Reflectance (SR) Product --- landsat-c1l2alb-sr',\n 'Landsat Collection 2 Level-2 Albers Top of Atmosphere (TA) Reflectance Product --- landsat-c2l2alb-ta',\n 'Landsat Collection 2 Level-2 Albers Surface Reflectance (SR) Product --- landsat-c2l2alb-sr',\n 'Landsat Collection 2 Analysis Ready Data (ARD) Level-2 UTM Top of Atmosphere (TA) Reflectance Product --- landsat-c2ard-ta',\n 'Landsat Collection 2 Level-3 Dynamic Surface Water Extent (DSWE) Product --- landsat-c2l3-dswe']\nlist(ls_cat.get_all_collections())\n\n[<CollectionClient id=landsat-c1l2alb-st>,\n <CollectionClient id=landsat-c2l2-sr>,\n <CollectionClient id=landsat-c2l2-st>,\n <CollectionClient id=landsat-c2ard-st>,\n <CollectionClient id=landsat-c2l2alb-bt>,\n <CollectionClient id=landsat-c2ard-bt>,\n <CollectionClient id=landsat-c2l3-fsca>,\n <CollectionClient id=landsat-c1l2alb-bt>,\n <CollectionClient id=landsat-c2l1>,\n <CollectionClient id=landsat-c2l3-ba>,\n <CollectionClient id=landsat-c1l2alb-ta>,\n <CollectionClient id=landsat-c2l2alb-st>,\n <CollectionClient id=landsat-c2ard-sr>,\n <CollectionClient id=landsat-c1l1>,\n <CollectionClient id=landsat-c1l2alb-sr>,\n <CollectionClient id=landsat-c2l2alb-ta>,\n <CollectionClient id=landsat-c2l2alb-sr>,\n <CollectionClient id=landsat-c2ard-ta>,\n <CollectionClient id=landsat-c2l3-dswe>]\nbbox = [-81.96624,26.38141,-81.80351,26.56457]\ndatetime_range = '2021-10-17T00:00:00Z/2021-10-31T23:59:59Z'\ndatetime_range\n\n'2021-10-17T00:00:00Z/2021-10-31T23:59:59Z'\nsearch = ls_cat.search(\n collections = ['landsat-c2l1'], # Landsat C2 L1 - Copied ID from above. Must be in a list\n bbox = bbox, # Specified above\n datetime = datetime_range # Specified above\n)\nsearch.matched()\n\n3\nitems = list(search.get_items())\nPrint first 5 item ids and associated datetime\n[f'{x.id} --- {x.datetime}' for x in items][:5]\n\n['LC08_L1TP_016042_20211027_20211104_02_T1 --- 2021-10-27 15:56:38.874611+00:00',\n 'LC08_L1TP_016041_20211027_20211104_02_T1 --- 2021-10-27 15:56:14.983570+00:00',\n 'LE07_L1TP_016042_20211019_20211114_02_T1 --- 2021-10-19 14:50:36.703778+00:00']\nhttp_links = []\n\nfor i in items:\n for a in i.assets.values():\n a_link = json.loads(json.dumps(a.to_dict()))\n if a_link['href'].endswith('.TIF'):\n http_links.append(a_link['href'])\nlen(http_links)\n\n49\nhttp_links[:10]\n\n['https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B1.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B2.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B3.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B4.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B5.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B6.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B7.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B8.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B9.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B10.TIF']\ns3_links = []\n\nfor i in items:\n for a in i.assets.values():\n a_link = json.loads(json.dumps(a.to_dict()))\n if 'alternate' in a_link:\n s3_asset = a_link['alternate']['s3']['href']\n if s3_asset.endswith('.TIF'):\n s3_links.append(s3_asset)\nlen(s3_links)\n\n49\ns3_links[:10]\n\n['s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B1.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B2.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B3.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B4.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B5.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B6.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B7.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B8.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B9.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B10.TIF']" + }, + { + "objectID": "examples/USGS_Landsat/Landsat_C2_Search_Access.html#data-access---rioxarray", + "href": "examples/USGS_Landsat/Landsat_C2_Search_Access.html#data-access---rioxarray", + "title": "USGS Landsat Cloud Data - Search and Access", + "section": "Data Access - rioxarray", + "text": "Data Access - rioxarray\nSet working environment\n\nsession = boto3.Session()\n\n\nrio_env = rio.Env(AWSSession(session, requester_pays=True), \n AWS_NO_SIGN_REQUEST='NO',\n GDAL_DISABLE_READDIR_ON_OPEN='TRUE')\nrio_env.__enter__()\n\n<rasterio.env.Env at 0x7f6e5276eb50>\n\n\n\ns3_url = 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B1.TIF'\n\nRead S3 asset as xarray datarray\n\nda = rioxarray.open_rasterio(s3_url, chunks='auto').squeeze('band', drop=True)\n\n\nda\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (y: 7821, x: 7661)>\ndask.array<getitem, shape=(7821, 7661), dtype=uint16, chunksize=(7821, 7661), chunktype=numpy.ndarray>\nCoordinates:\n * x (x) float64 2.889e+05 2.889e+05 ... 5.187e+05 5.187e+05\n * y (y) float64 2.992e+06 2.992e+06 ... 2.758e+06 2.758e+06\n spatial_ref int64 0\nAttributes:\n _FillValue: 0.0\n scale_factor: 1.0\n add_offset: 0.0xarray.DataArrayy: 7821x: 7661dask.array<chunksize=(7821, 7661), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n114.28 MiB\n114.28 MiB\n\n\nShape\n(7821, 7661)\n(7821, 7661)\n\n\nCount\n3 Tasks\n1 Chunks\n\n\nType\nuint16\nnumpy.ndarray\n\n\n\n\n\n\n\nCoordinates: (3)x(x)float642.889e+05 2.889e+05 ... 5.187e+05array([288900., 288930., 288960., ..., 518640., 518670., 518700.])y(y)float642.992e+06 2.992e+06 ... 2.758e+06array([2992200., 2992170., 2992140., ..., 2757660., 2757630., 2757600.])spatial_ref()int640crs_wkt :PROJCS[\"WGS 84 / UTM zone 17N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-81],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32617\"]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :WGS 84horizontal_datum_name :World Geodetic System 1984projected_crs_name :WGS 84 / UTM zone 17Ngrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-81.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"WGS 84 / UTM zone 17N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-81],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32617\"]]GeoTransform :288885.0 30.0 0.0 2992215.0 0.0 -30.0array(0)Attributes: (3)_FillValue :0.0scale_factor :1.0add_offset :0.0\n\n\nPlot scene\n\n#da.hvplot(x='x', y='y', crs='epsg:32617', rasterize=True, width=800, height=600, tiles = 'ESRI') # TODO: how to exclude fill values\nda.hvplot(x='x', y='y', crs='epsg:32617', rasterize=True, width=800, height=600)" + }, + { + "objectID": "examples/USGS_Landsat/Landsat_C2_Search_Access.html#aws-cli", + "href": "examples/USGS_Landsat/Landsat_C2_Search_Access.html#aws-cli", + "title": "USGS Landsat Cloud Data - Search and Access", + "section": "AWS CLI", + "text": "AWS CLI\nList content in bucket\n\n! aws s3 ls s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/ --request-payer requester\n\n2021-11-05 08:37:33 116979 LC08_L1TP_016042_20211027_20211104_02_T1_ANG.txt\n2021-11-05 08:37:33 77713726 LC08_L1TP_016042_20211027_20211104_02_T1_B1.TIF\n2021-11-05 08:37:34 76108899 LC08_L1TP_016042_20211027_20211104_02_T1_B10.TIF\n2021-11-05 08:37:36 75282069 LC08_L1TP_016042_20211027_20211104_02_T1_B11.TIF\n2021-11-05 08:37:37 77374352 LC08_L1TP_016042_20211027_20211104_02_T1_B2.TIF\n2021-11-05 08:37:38 77527685 LC08_L1TP_016042_20211027_20211104_02_T1_B3.TIF\n2021-11-05 08:37:39 76485245 LC08_L1TP_016042_20211027_20211104_02_T1_B4.TIF\n2021-11-05 08:37:40 78304185 LC08_L1TP_016042_20211027_20211104_02_T1_B5.TIF\n2021-11-05 08:37:42 75847018 LC08_L1TP_016042_20211027_20211104_02_T1_B6.TIF\n2021-11-05 08:37:44 73697164 LC08_L1TP_016042_20211027_20211104_02_T1_B7.TIF\n2021-11-05 08:37:45 304168225 LC08_L1TP_016042_20211027_20211104_02_T1_B8.TIF\n2021-11-05 08:37:48 60967770 LC08_L1TP_016042_20211027_20211104_02_T1_B9.TIF\n2021-11-05 08:37:49 15036 LC08_L1TP_016042_20211027_20211104_02_T1_MTL.json\n2021-11-05 08:37:49 12369 LC08_L1TP_016042_20211027_20211104_02_T1_MTL.txt\n2021-11-05 08:37:49 18065 LC08_L1TP_016042_20211027_20211104_02_T1_MTL.xml\n2021-11-05 08:37:49 3527153 LC08_L1TP_016042_20211027_20211104_02_T1_QA_PIXEL.TIF\n2021-11-05 08:37:49 222983 LC08_L1TP_016042_20211027_20211104_02_T1_QA_RADSAT.TIF\n2021-11-05 08:37:49 2052489 LC08_L1TP_016042_20211027_20211104_02_T1_SAA.TIF\n2021-11-05 08:37:49 1783702 LC08_L1TP_016042_20211027_20211104_02_T1_SZA.TIF\n2021-11-05 08:37:50 8239344 LC08_L1TP_016042_20211027_20211104_02_T1_VAA.TIF\n2021-11-05 08:37:50 2930826 LC08_L1TP_016042_20211027_20211104_02_T1_VZA.TIF\n2021-11-05 08:37:50 26399 LC08_L1TP_016042_20211027_20211104_02_T1_stac.json\n2021-11-05 08:37:50 46747 LC08_L1TP_016042_20211027_20211104_02_T1_thumb_large.jpeg\n2021-11-05 08:37:50 6015 LC08_L1TP_016042_20211027_20211104_02_T1_thumb_small.jpeg\n\n\nDownload content in bucket to local\n\n! aws s3 cp s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/ . --recursive --request-payer \n\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_ANG.txt to ./LC08_L1TP_016042_20211027_20211104_02_T1_ANG.txt\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B10.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B10.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B2.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B2.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B1.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B1.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B11.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B11.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B6.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B6.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B3.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B3.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B4.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B4.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B5.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B5.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_MTL.json to ./LC08_L1TP_016042_20211027_20211104_02_T1_MTL.json\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_MTL.txt to ./LC08_L1TP_016042_20211027_20211104_02_T1_MTL.txt\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_MTL.xml to ./LC08_L1TP_016042_20211027_20211104_02_T1_MTL.xml\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B7.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B7.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_SZA.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_SZA.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_SAA.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_SAA.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_QA_RADSAT.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_QA_RADSAT.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_QA_PIXEL.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_QA_PIXEL.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_VAA.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_VAA.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_stac.json to ./LC08_L1TP_016042_20211027_20211104_02_T1_stac.json\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_thumb_large.jpeg to ./LC08_L1TP_016042_20211027_20211104_02_T1_thumb_large.jpeg\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_thumb_small.jpeg to ./LC08_L1TP_016042_20211027_20211104_02_T1_thumb_small.jpeg\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B9.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B9.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_VZA.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_VZA.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B8.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B8.TIF" + }, + { + "objectID": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html", + "href": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html", + "title": "Searching and Downloading NSIDC Cloud Collections", + "section": "", + "text": "Programmatic access and processing of NSIDC data can happen in 2 ways, using the old Search -> Download -> Analize pattern or using a more modern Search -> Process_in_the_cloud -> Analyze approach.\nThere is nothing wrong with downloading data to our local machine but that can get complicated or even impossible if a dataset is too large. For this reason NSIDC along with other NASA data centers started to collocate or migrate their dataset holdings to the cloud.\nIn order to use NSIDC cloud collections we need to 1. Authenticate ourselves with the NASA Earthdata Login API (EDL). 2. Search granules/collections using a CMR client that supports authentication 3. Parse CMR responses looking for AWS S3 URLs 4. Access the data granules using temporary AWS credentials given by the NSIDC cloud credentials endpoint" + }, + { + "objectID": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#data-used", + "href": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#data-used", + "title": "Searching and Downloading NSIDC Cloud Collections", + "section": "Data used:", + "text": "Data used:\n\nICESat-2 ATL03: This data set contains height above the WGS 84 ellipsoid (ITRF2014 reference frame), latitude, longitude, and time for all photons." + }, + { + "objectID": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#requirements", + "href": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#requirements", + "title": "Searching and Downloading NSIDC Cloud Collections", + "section": "Requirements", + "text": "Requirements\n\nNASA Eartdata Login (EDL) credentials\npython libraries:\n\naws-cli\nxarray\nfs-s3fs\nzarr\ncmr\n\npython-cmr (optional fallback) NSIDC fork" + }, + { + "objectID": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#querying-cmr-for-nsidc-data", + "href": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#querying-cmr-for-nsidc-data", + "title": "Searching and Downloading NSIDC Cloud Collections", + "section": "Querying CMR for NSIDC data", + "text": "Querying CMR for NSIDC data\nMost collections at NSIDC have not being migrated to the cloud and can be found using CMR with no authentication at all. Here is a simple example for altimeter data (ATL03) coming from the ICESat-2 mission. First we’ll search the regular collection and then we’ll do the same using the cloud collection.\nNote: This notebook uses a low level CMR endpoint, this won’t be not the only workflow for data discovery.\n\nfrom cmr.search import collection as cmr_collection\nfrom cmr.search import granule \nfrom cmr.auth import token\n\nimport textwrap\n# NON_AWS collections are hosted at the NSIDC DAAC data center\n# AWS_CLOUD collections are hosted at AWS S3 us-west-2\nNSIDC_PROVIDERS = {\n 'NSIDC_HOSTED': 'NSIDC_ECS', \n 'AWS_HOSTED':'NSIDC_CPRD'\n}\n\n# First let's search for some collections hosted at NSIDC using a keyword\ncollections = cmr_collection.search({'keyword':'ice',\n 'provider': NSIDC_PROVIDERS['NSIDC_HOSTED']})\n\n# Let's print some information about the first 3 collection that match our provider\nfor collection in collections[0:3]:\n wrapped_abstract = '\\n'.join(textwrap.wrap(f\"Abstract: {collection['umm']['Abstract']}\", 80)) + '\\n'\n print(f\"concept-id: {collection['meta']['concept-id']}\\n\" +\n f\"Title: {collection['umm']['EntryTitle']}\\n\" +\n wrapped_abstract)\n\nconcept-id: C1997321091-NSIDC_ECS\nTitle: ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004\nAbstract: This data set (ATL03) contains height above the WGS 84 ellipsoid\n(ITRF2014 reference frame), latitude, longitude, and time for all photons\ndownlinked by the Advanced Topographic Laser Altimeter System (ATLAS) instrument\non board the Ice, Cloud and land Elevation Satellite-2 (ICESat-2) observatory.\nThe ATL03 product was designed to be a single source for all photon data and\nancillary information needed by higher-level ATLAS/ICESat-2 products. As such,\nit also includes spacecraft and instrument parameters and ancillary data not\nexplicitly required for ATL03.\n\nconcept-id: C1705401930-NSIDC_ECS\nTitle: ATLAS/ICESat-2 L2A Global Geolocated Photon Data V003\nAbstract: This data set (ATL03) contains height above the WGS 84 ellipsoid\n(ITRF2014 reference frame), latitude, longitude, and time for all photons\ndownlinked by the Advanced Topographic Laser Altimeter System (ATLAS) instrument\non board the Ice, Cloud and land Elevation Satellite-2 (ICESat-2) observatory.\nThe ATL03 product was designed to be a single source for all photon data and\nancillary information needed by higher-level ATLAS/ICESat-2 products. As such,\nit also includes spacecraft and instrument parameters and ancillary data not\nexplicitly required for ATL03.\n\nconcept-id: C2003771331-NSIDC_ECS\nTitle: ATLAS/ICESat-2 L3A Land Ice Height V004\nAbstract: This data set (ATL06) provides geolocated, land-ice surface heights\n(above the WGS 84 ellipsoid, ITRF2014 reference frame), plus ancillary\nparameters that can be used to interpret and assess the quality of the height\nestimates. The data were acquired by the Advanced Topographic Laser Altimeter\nSystem (ATLAS) instrument on board the Ice, Cloud and land Elevation Satellite-2\n(ICESat-2) observatory.\n\n\n\n\n# Now let's do the same with short names, a more specific way of finding data.\n\n#First let's search for some collections hosted at NSIDC\ncollections = cmr_collection.search({'short_name':'ATL03',\n 'provider': NSIDC_PROVIDERS['NSIDC_HOSTED']})\n\n# Note how we get back the same collection twice, that's because we have 2 versions available.\nfor collection in collections[0:3]:\n wrapped_abstract = '\\n'.join(textwrap.wrap(f\"Abstract: {collection['umm']['Abstract']}\", 80)) + '\\n'\n print(f\"concept-id: {collection['meta']['concept-id']}\\n\" +\n f\"Title: {collection['umm']['EntryTitle']}\\n\" +\n wrapped_abstract)\n\nconcept-id: C1997321091-NSIDC_ECS\nTitle: ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004\nAbstract: This data set (ATL03) contains height above the WGS 84 ellipsoid\n(ITRF2014 reference frame), latitude, longitude, and time for all photons\ndownlinked by the Advanced Topographic Laser Altimeter System (ATLAS) instrument\non board the Ice, Cloud and land Elevation Satellite-2 (ICESat-2) observatory.\nThe ATL03 product was designed to be a single source for all photon data and\nancillary information needed by higher-level ATLAS/ICESat-2 products. As such,\nit also includes spacecraft and instrument parameters and ancillary data not\nexplicitly required for ATL03.\n\nconcept-id: C1705401930-NSIDC_ECS\nTitle: ATLAS/ICESat-2 L2A Global Geolocated Photon Data V003\nAbstract: This data set (ATL03) contains height above the WGS 84 ellipsoid\n(ITRF2014 reference frame), latitude, longitude, and time for all photons\ndownlinked by the Advanced Topographic Laser Altimeter System (ATLAS) instrument\non board the Ice, Cloud and land Elevation Satellite-2 (ICESat-2) observatory.\nThe ATL03 product was designed to be a single source for all photon data and\nancillary information needed by higher-level ATLAS/ICESat-2 products. As such,\nit also includes spacecraft and instrument parameters and ancillary data not\nexplicitly required for ATL03.\n\n\n\n\n# now that we have the concept-ids we can look for data granules in that collection and pass spatiotemporal parameters.\nfrom cmr_serializer import QueryResult\n\n# a bbox over Juneau Icefield \n# bbox = min Longitude , min Latitude , max Longitude , max Latitude \nquery = {'concept-id': 'C1997321091-NSIDC_ECS',\n 'bounding_box': '-135.1977,58.3325,-133.3410,58.9839'}\n\n# Querying for ATL03 v3 using its concept-id and a bounding box\nresults = granule.search(query, limit=1000)\n# This is a wrapper with convenient methods to work with CMR query results.\ngranules = QueryResult(results).items()\n\nprint(f\"Total granules found: {len(results)} \\n\")\nfor g in granules[0:3]:\n display(g)\n\nTotal granules found: 201 \n\n\n\n\n \n Id: ATL03_20181014001049_02350102_004_01.h5\n Collection: {'EntryTitle': 'ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004'}\n Spatial coverage: {'HorizontalSpatialDomain': {'Orbit': {'AscendingCrossing': -127.0482205607256, 'StartLatitude': 27.0, 'StartDirection': 'A', 'EndLatitude': 59.5, 'EndDirection': 'A'}}}\n Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2018-10-14T00:10:49.722Z', 'EndingDateTime': '2018-10-14T00:19:19.918Z'}}\n Size(MB): 1764.5729866028 \n Data: https://n5eil01u.ecs.nsidc.org/DP9/ATLAS/ATL03.004/2018.10.14/ATL03_20181014001049_02350102_004_01.h5\n \n \n \n\n\n\n \n Id: ATL03_20181015124359_02580106_004_01.h5\n Collection: {'EntryTitle': 'ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004'}\n Spatial coverage: {'HorizontalSpatialDomain': {'Orbit': {'AscendingCrossing': 49.70324528818096, 'StartLatitude': 59.5, 'StartDirection': 'D', 'EndLatitude': 27.0, 'EndDirection': 'D'}}}\n Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2018-10-15T12:43:57.696Z', 'EndingDateTime': '2018-10-15T12:52:28.274Z'}}\n Size(MB): 276.2403841019 \n Data: https://n5eil01u.ecs.nsidc.org/DP9/ATLAS/ATL03.004/2018.10.15/ATL03_20181015124359_02580106_004_01.h5\n \n \n \n\n\n\n \n Id: ATL03_20181018000228_02960102_004_01.h5\n Collection: {'EntryTitle': 'ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004'}\n Spatial coverage: {'HorizontalSpatialDomain': {'Orbit': {'AscendingCrossing': -127.82682215638665, 'StartLatitude': 27.0, 'StartDirection': 'A', 'EndLatitude': 59.5, 'EndDirection': 'A'}}}\n Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2018-10-18T00:02:28.717Z', 'EndingDateTime': '2018-10-18T00:10:58.903Z'}}\n Size(MB): 877.0574979782 \n Data: https://n5eil01u.ecs.nsidc.org/DP9/ATLAS/ATL03.004/2018.10.18/ATL03_20181018000228_02960102_004_01.h5\n \n \n \n\n\n\n# We can access the data links with the data_links()\nfor g in granules[0:10]:\n print(g.data_links())" + }, + { + "objectID": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#cloud-collections", + "href": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#cloud-collections", + "title": "Searching and Downloading NSIDC Cloud Collections", + "section": "Cloud Collections", + "text": "Cloud Collections\nSome NSIDC cloud collections are not yet public we need to authenticate ourselves with CMR first.\n\nimport getpass\nimport textwrap\n\nfrom cmr.search import collection as cmr_collection\nfrom cmr.search import granule \nfrom cmr.auth import token\n\nfrom cmr_auth import CMRAuth\n\n# NON_AWS collections are hosted at the NSIDC DAAC data center\n# AWS_CLOUD collections are hosted at AWS S3 us-west-2\nNSIDC_PROVIDERS = {\n 'NSIDC_HOSTED': 'NSIDC_ECS', \n 'AWS_HOSTED':'NSIDC_CPRD'\n}\n\n# Use your own EDL username\nUSER= 'betolink'\n\nprint('Enter your NASA Earthdata login password:')\npassword = getpass.getpass()\nCMR_auth = CMRAuth(USER, password)\n# Token to search private collections on CMR\ncmr_token = CMR_auth.get_token()\n\nEnter your NASA Earthdata login password:\n\n\n ········\n\n\n\n# Now let's start our aunthenticated queries on CMR\nquery = {'short_name':'ATL03',\n 'token': cmr_token,\n 'provider': NSIDC_PROVIDERS['AWS_HOSTED']}\n\ncollections = cmr_collection.search(query)\n\nfor collection in collections[0:3]:\n wrapped_abstract = '\\n'.join(textwrap.wrap(f\"Abstract: {collection['umm']['Abstract']}\", 80)) + '\\n'\n print(f\"concept-id: {collection['meta']['concept-id']}\\n\" +\n f\"Title: {collection['umm']['EntryTitle']}\\n\" +\n wrapped_abstract)\n\nconcept-id: C2027878642-NSIDC_CPRD\nTitle: ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004\nAbstract: This data set (ATL03) contains height above the WGS 84 ellipsoid\n(ITRF2014 reference frame), latitude, longitude, and time for all photons\ndownlinked by the Advanced Topographic Laser Altimeter System (ATLAS) instrument\non board the Ice, Cloud and land Elevation Satellite-2 (ICESat-2) observatory.\nThe ATL03 product was designed to be a single source for all photon data and\nancillary information needed by higher-level ATLAS/ICESat-2 products. As such,\nit also includes spacecraft and instrument parameters and ancillary data not\nexplicitly required for ATL03.\n\n\n\n\n# now that we have the concept-id for our ATL03 in the cloud we do the same thing we did with ATL03 hosted at\nfrom cmr_serializer import QueryResult\n# NSIDC but using the cloud concept-id\n# Jeneau ice sheet\nquery = {'concept-id': 'C2027878642-NSIDC_CPRD',\n 'token': cmr_token,\n 'bounding_box': '-135.1977,58.3325,-133.3410,58.9839'}\n\n# Querying for ATL03 v3 using its concept-id and a bounding box\nresults = granule.search(query, limit=1000)\ngranules = QueryResult(results).items()\n\nprint(f\"Total granules found: {len(results)} \\n\")\n\n# Print the first 3 granules\nfor g in granules[0:3]:\n display(g)\n # You can use: print(g) for the regular text representation.\n\nTotal granules found: 135 \n\n\n\n\n \n Id: ATL03_20181014001049_02350102_004_01.h5\n Collection: {'EntryTitle': 'ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004'}\n Spatial coverage: {'HorizontalSpatialDomain': {'Orbit': {'AscendingCrossing': -127.0482205607256, 'StartLatitude': 27.0, 'StartDirection': 'A', 'EndLatitude': 59.5, 'EndDirection': 'A'}}}\n Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2018-10-14T00:10:49.722Z', 'EndingDateTime': '2018-10-14T00:19:19.918Z'}}\n Size(MB): 1764.5729866027832 \n Data: https://data.nsidc.earthdatacloud.nasa.gov/nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/14/ATL03_20181014001049_02350102_004_01.h5s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/14/ATL03_20181014001049_02350102_004_01.h5\n \n \n \n\n\n\n \n Id: ATL03_20181015124359_02580106_004_01.h5\n Collection: {'EntryTitle': 'ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004'}\n Spatial coverage: {'HorizontalSpatialDomain': {'Orbit': {'AscendingCrossing': 49.70324528818096, 'StartLatitude': 59.5, 'StartDirection': 'D', 'EndLatitude': 27.0, 'EndDirection': 'D'}}}\n Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2018-10-15T12:43:57.696Z', 'EndingDateTime': '2018-10-15T12:52:28.274Z'}}\n Size(MB): 276.2403841018677 \n Data: https://data.nsidc.earthdatacloud.nasa.gov/nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/15/ATL03_20181015124359_02580106_004_01.h5s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/15/ATL03_20181015124359_02580106_004_01.h5\n \n \n \n\n\n\n \n Id: ATL03_20181018000228_02960102_004_01.h5\n Collection: {'EntryTitle': 'ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004'}\n Spatial coverage: {'HorizontalSpatialDomain': {'Orbit': {'AscendingCrossing': -127.82682215638665, 'StartLatitude': 27.0, 'StartDirection': 'A', 'EndLatitude': 59.5, 'EndDirection': 'A'}}}\n Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2018-10-18T00:02:28.717Z', 'EndingDateTime': '2018-10-18T00:10:58.903Z'}}\n Size(MB): 877.0574979782104 \n Data: https://data.nsidc.earthdatacloud.nasa.gov/nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/18/ATL03_20181018000228_02960102_004_01.h5s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/18/ATL03_20181018000228_02960102_004_01.h5\n \n \n \n\n\n\nNOTE: Not all the data granules for NSIDC datasets have been migrated to S3. This might result in different counts between the NSIDC hosted data collections and the ones in AWS S3\n\n# We can list the s3 links but \nfor g in granules[0:10]:\n print(g.data_links(only_s3=True))\n\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/14/ATL03_20181014001049_02350102_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/15/ATL03_20181015124359_02580106_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/18/ATL03_20181018000228_02960102_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/11/05/ATL03_20181105113651_05780106_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/11/07/ATL03_20181107225525_06160102_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/11/09/ATL03_20181109112837_06390106_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/11/11/ATL03_20181111224708_06770102_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/11/15/ATL03_20181115223845_07380102_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/12/04/ATL03_20181204101243_10200106_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/12/06/ATL03_20181206213114_10580102_004_01.h5']\n\n\nWe note that our RelatedLinks array now contain links to AWS S3, these are the direct URIs for our data granules in the AWS us-west-2 region." + }, + { + "objectID": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#data-access-using-aws-s3", + "href": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#data-access-using-aws-s3", + "title": "Searching and Downloading NSIDC Cloud Collections", + "section": "Data Access using AWS S3", + "text": "Data Access using AWS S3\n\nIMPORTANT: This section will only work if this notebook is running on the AWS us-west-2 zone\n\nThere is more than one way of accessing data on AWS S3, either downloading it to your local machine using the official client library or using a python library.\nPerformance tip: using the HTTPS URLs will decrease the access performance since these links have to internally be processed by AWS’s content delivery system (CloudFront). To get a better performance we should access the S3:// URLs with BOTO3 or a high level S3 enabled library (i.e. S3FS)\nRelated links: * HDF in the Cloud challenges and solutions for scientific data * Cloud Storage (Amazon S3) HDF5 Connector\n\n# READ only temporary credentials\nimport s3fs\nimport h5py\n\n# This credentials only last 1 hour.\ns3_cred = CMR_auth.get_s3_credentials()\n\n\ns3_fs = s3fs.S3FileSystem(key=s3_cred['accessKeyId'],\n secret=s3_cred['secretAccessKey'],\n token=s3_cred['sessionToken'])\n\n# Now you could grab S3 links to your cloud instance (EC2, Hub etc) using:\n# s3_fs.get('s3://SOME_LOCATION/ATL03_20181015124359_02580106_004_01.h5', 'test.h5')\n\n\nWe now have the propper credentials and file mapper to access the data within AWS us-west-2.\n\nwith s3_fs.open('s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/15/ATL03_20181015124359_02580106_004_01.h5', 'rb') as s3f:\n with h5py.File(s3f, 'r') as f:\n print([key for key in f.keys()])\n\n\n\nUsing xarray to open files on S3\nATL data is complex so xarray doesn’t know how to extract the important bits out of it.\n\nimport xarray\n\nwith s3_fs.open('s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/15/ATL03_20181015124359_02580106_004_01.h5', 'rb') as s3f:\n ds= xarray.open_dataset(s3f)\n for varname in ds:\n print(varname)\nds\n\n\n\n“Downloading” files on S3 using the official aws-cli library\nThe quotes on downloading are because ideally you’ll be working on an EC2 (virtual machine for short) instance on the us-west-2 region." + }, + { + "objectID": "examples/ORNL/Data_Access__Direct_S3_Access__ORNL_DAYMET.html#single-file-in-region-direct-s3-access-of-netcdf-file", + "href": "examples/ORNL/Data_Access__Direct_S3_Access__ORNL_DAYMET.html#single-file-in-region-direct-s3-access-of-netcdf-file", + "title": "Direct S3 Data Access: NetCDF - Daymet v4 Daily TMAX Example", + "section": "Single file in-region direct S3 access of netcdf file", + "text": "Single file in-region direct S3 access of netcdf file\n\nfs_s3 = s3fs.S3FileSystem(anon=False, key=temp_creds_req['accessKeyId'], secret=temp_creds_req['secretAccessKey'], token=temp_creds_req['sessionToken'])\n\n\ns3_url = s3_urls[0]\ns3_url\n\n's3://ornl-cumulus-prod-protected/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_tmax_2015.nc'\n\n\n\ns3_file_obj = fs_s3.open(s3_url, mode='rb')\n\n\nxr_ds = xr.open_dataset(s3_file_obj, chunks='auto', engine='h5netcdf')\nxr_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (x: 7814, y: 8075, time: 365, nv: 2)\nCoordinates:\n * x (x) float32 -4.56e+06 -4.559e+06 ... 3.253e+06\n * y (y) float32 4.984e+06 4.983e+06 ... -3.09e+06\n lat (y, x) float32 dask.array<chunksize=(8075, 3907), meta=np.ndarray>\n lon (y, x) float32 dask.array<chunksize=(8075, 3907), meta=np.ndarray>\n * time (time) datetime64[ns] 2015-01-01T12:00:00 ... 20...\nDimensions without coordinates: nv\nData variables:\n yearday (time) int16 dask.array<chunksize=(365,), meta=np.ndarray>\n time_bnds (time, nv) datetime64[ns] dask.array<chunksize=(365, 2), meta=np.ndarray>\n lambert_conformal_conic int16 ...\n tmax (time, y, x) float32 dask.array<chunksize=(55, 475, 1282), meta=np.ndarray>\nAttributes:\n start_year: 2015\n source: Daymet Software Version 4.0\n Version_software: Daymet Software Version 4.0\n Version_data: Daymet Data Version 4.0\n Conventions: CF-1.6\n citation: Please see http://daymet.ornl.gov/ for current Daymet ...\n references: Please see http://daymet.ornl.gov/ for current informa...xarray.DatasetDimensions:x: 7814y: 8075time: 365nv: 2Coordinates: (5)x(x)float32-4.56e+06 -4.559e+06 ... 3.253e+06units :mlong_name :x coordinate of projectionstandard_name :projection_x_coordinatearray([-4560250., -4559250., -4558250., ..., 3250750., 3251750., 3252750.],\n dtype=float32)y(y)float324.984e+06 4.983e+06 ... -3.09e+06units :mlong_name :y coordinate of projectionstandard_name :projection_y_coordinatearray([ 4984000., 4983000., 4982000., ..., -3088000., -3089000., -3090000.],\n dtype=float32)lat(y, x)float32dask.array<chunksize=(8075, 3907), meta=np.ndarray>units :degrees_northlong_name :latitude coordinatestandard_name :latitude\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n240.70 MiB\n120.35 MiB\n\n\nShape\n(8075, 7814)\n(8075, 3907)\n\n\nCount\n3 Tasks\n2 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nlon\n\n\n(y, x)\n\n\nfloat32\n\n\ndask.array<chunksize=(8075, 3907), meta=np.ndarray>\n\n\n\n\nunits :\n\ndegrees_east\n\nlong_name :\n\nlongitude coordinate\n\nstandard_name :\n\nlongitude\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n240.70 MiB\n120.35 MiB\n\n\nShape\n(8075, 7814)\n(8075, 3907)\n\n\nCount\n3 Tasks\n2 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\ntime\n\n\n(time)\n\n\ndatetime64[ns]\n\n\n2015-01-01T12:00:00 ... 2015-12-...\n\n\n\n\nstandard_name :\n\ntime\n\nbounds :\n\ntime_bnds\n\nlong_name :\n\n24-hour day based on local time\n\n\n\n\narray(['2015-01-01T12:00:00.000000000', '2015-01-02T12:00:00.000000000',\n '2015-01-03T12:00:00.000000000', ..., '2015-12-29T12:00:00.000000000',\n '2015-12-30T12:00:00.000000000', '2015-12-31T12:00:00.000000000'],\n dtype='datetime64[ns]')\n\nData variables: (4)\n\n\n\n\n\nyearday\n\n\n(time)\n\n\nint16\n\n\ndask.array<chunksize=(365,), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nday of year (DOY) starting with day 1 on Januaray 1st\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n730 B\n730 B\n\n\nShape\n(365,)\n(365,)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nint16\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\ntime_bnds\n\n\n(time, nv)\n\n\ndatetime64[ns]\n\n\ndask.array<chunksize=(365, 2), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n5.70 kiB\n5.70 kiB\n\n\nShape\n(365, 2)\n(365, 2)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\ndatetime64[ns]\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nlambert_conformal_conic\n\n\n()\n\n\nint16\n\n\n...\n\n\n\n\ngrid_mapping_name :\n\nlambert_conformal_conic\n\nlongitude_of_central_meridian :\n\n-100.0\n\nlatitude_of_projection_origin :\n\n42.5\n\nfalse_easting :\n\n0.0\n\nfalse_northing :\n\n0.0\n\nstandard_parallel :\n\n[25. 60.]\n\nsemi_major_axis :\n\n6378137.0\n\ninverse_flattening :\n\n298.257223563\n\n\n\n\narray(-32767, dtype=int16)\n\n\ntmax\n\n\n(time, y, x)\n\n\nfloat32\n\n\ndask.array<chunksize=(55, 475, 1282), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ndaily maximum temperature\n\nunits :\n\ndegrees C\n\ngrid_mapping :\n\nlambert_conformal_conic\n\ncell_methods :\n\narea: mean time: maximum\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n85.80 GiB\n127.76 MiB\n\n\nShape\n(365, 8075, 7814)\n(55, 475, 1282)\n\n\nCount\n834 Tasks\n833 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nAttributes: (7)start_year :2015source :Daymet Software Version 4.0Version_software :Daymet Software Version 4.0Version_data :Daymet Data Version 4.0Conventions :CF-1.6citation :Please see http://daymet.ornl.gov/ for current Daymet data citation informationreferences :Please see http://daymet.ornl.gov/ for current information on Daymet references" + }, + { + "objectID": "examples/ORNL/Data_Access__Direct_S3_Access__ORNL_DAYMET.html#multi-file-in-region-direct-s3-access-of-netcdf-files", + "href": "examples/ORNL/Data_Access__Direct_S3_Access__ORNL_DAYMET.html#multi-file-in-region-direct-s3-access-of-netcdf-files", + "title": "Direct S3 Data Access: NetCDF - Daymet v4 Daily TMAX Example", + "section": "Multi-file in-region direct S3 access of netcdf files", + "text": "Multi-file in-region direct S3 access of netcdf files\n\n# Iterate through remote_files to create a fileset\nfileset = [fs_s3.open(file) for file in s3_urls]\n\n\n# This works...if you rerun this line and get a context manager error, try 1. rerunning the line above then this line \nxr_ts = xr.open_mfdataset(fileset, chunks='auto', engine='h5netcdf')\n\n\nxr_ts\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (x: 7814, y: 8075, time: 2190, nv: 2)\nCoordinates:\n * x (x) float32 -4.56e+06 -4.559e+06 ... 3.253e+06\n * y (y) float32 4.984e+06 4.983e+06 ... -3.09e+06\n lat (y, x) float32 dask.array<chunksize=(8075, 3907), meta=np.ndarray>\n lon (y, x) float32 dask.array<chunksize=(8075, 3907), meta=np.ndarray>\n * time (time) datetime64[ns] 2015-01-01T12:00:00 ... 20...\nDimensions without coordinates: nv\nData variables:\n yearday (time) int16 dask.array<chunksize=(365,), meta=np.ndarray>\n time_bnds (time, nv) datetime64[ns] dask.array<chunksize=(365, 2), meta=np.ndarray>\n lambert_conformal_conic (time) int16 -32767 -32767 -32767 ... -32767 -32767\n tmax (time, y, x) float32 dask.array<chunksize=(55, 475, 1282), meta=np.ndarray>\nAttributes:\n start_year: 2015\n source: Daymet Software Version 4.0\n Version_software: Daymet Software Version 4.0\n Version_data: Daymet Data Version 4.0\n Conventions: CF-1.6\n citation: Please see http://daymet.ornl.gov/ for current Daymet ...\n references: Please see http://daymet.ornl.gov/ for current informa...xarray.DatasetDimensions:x: 7814y: 8075time: 2190nv: 2Coordinates: (5)x(x)float32-4.56e+06 -4.559e+06 ... 3.253e+06units :mlong_name :x coordinate of projectionstandard_name :projection_x_coordinatearray([-4560250., -4559250., -4558250., ..., 3250750., 3251750., 3252750.],\n dtype=float32)y(y)float324.984e+06 4.983e+06 ... -3.09e+06units :mlong_name :y coordinate of projectionstandard_name :projection_y_coordinatearray([ 4984000., 4983000., 4982000., ..., -3088000., -3089000., -3090000.],\n dtype=float32)lat(y, x)float32dask.array<chunksize=(8075, 3907), meta=np.ndarray>units :degrees_northlong_name :latitude coordinatestandard_name :latitude\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n240.70 MiB\n120.35 MiB\n\n\nShape\n(8075, 7814)\n(8075, 3907)\n\n\nCount\n51 Tasks\n2 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nlon\n\n\n(y, x)\n\n\nfloat32\n\n\ndask.array<chunksize=(8075, 3907), meta=np.ndarray>\n\n\n\n\nunits :\n\ndegrees_east\n\nlong_name :\n\nlongitude coordinate\n\nstandard_name :\n\nlongitude\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n240.70 MiB\n120.35 MiB\n\n\nShape\n(8075, 7814)\n(8075, 3907)\n\n\nCount\n51 Tasks\n2 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\ntime\n\n\n(time)\n\n\ndatetime64[ns]\n\n\n2015-01-01T12:00:00 ... 2020-12-...\n\n\n\n\nstandard_name :\n\ntime\n\nbounds :\n\ntime_bnds\n\nlong_name :\n\n24-hour day based on local time\n\n\n\n\narray(['2015-01-01T12:00:00.000000000', '2015-01-02T12:00:00.000000000',\n '2015-01-03T12:00:00.000000000', ..., '2020-12-28T12:00:00.000000000',\n '2020-12-29T12:00:00.000000000', '2020-12-30T12:00:00.000000000'],\n dtype='datetime64[ns]')\n\nData variables: (4)\n\n\n\n\n\nyearday\n\n\n(time)\n\n\nint16\n\n\ndask.array<chunksize=(365,), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nday of year (DOY) starting with day 1 on Januaray 1st\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n4.28 kiB\n730 B\n\n\nShape\n(2190,)\n(365,)\n\n\nCount\n18 Tasks\n6 Chunks\n\n\nType\nint16\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\ntime_bnds\n\n\n(time, nv)\n\n\ndatetime64[ns]\n\n\ndask.array<chunksize=(365, 2), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n34.22 kiB\n5.70 kiB\n\n\nShape\n(2190, 2)\n(365, 2)\n\n\nCount\n18 Tasks\n6 Chunks\n\n\nType\ndatetime64[ns]\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nlambert_conformal_conic\n\n\n(time)\n\n\nint16\n\n\n-32767 -32767 ... -32767 -32767\n\n\n\n\ngrid_mapping_name :\n\nlambert_conformal_conic\n\nlongitude_of_central_meridian :\n\n-100.0\n\nlatitude_of_projection_origin :\n\n42.5\n\nfalse_easting :\n\n0.0\n\nfalse_northing :\n\n0.0\n\nstandard_parallel :\n\n[25. 60.]\n\nsemi_major_axis :\n\n6378137.0\n\ninverse_flattening :\n\n298.257223563\n\n\n\n\narray([-32767, -32767, -32767, ..., -32767, -32767, -32767], dtype=int16)\n\n\ntmax\n\n\n(time, y, x)\n\n\nfloat32\n\n\ndask.array<chunksize=(55, 475, 1282), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ndaily maximum temperature\n\nunits :\n\ndegrees C\n\ngrid_mapping :\n\nlambert_conformal_conic\n\ncell_methods :\n\narea: mean time: maximum\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n514.78 GiB\n127.76 MiB\n\n\nShape\n(2190, 8075, 7814)\n(55, 475, 1282)\n\n\nCount\n10002 Tasks\n4998 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nAttributes: (7)start_year :2015source :Daymet Software Version 4.0Version_software :Daymet Software Version 4.0Version_data :Daymet Data Version 4.0Conventions :CF-1.6citation :Please see http://daymet.ornl.gov/ for current Daymet data citation informationreferences :Please see http://daymet.ornl.gov/ for current information on Daymet references\n\n\n\n#xr_ts.SSH.hvplot.image()" + }, + { + "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html", + "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html", + "title": "PO.DAAC ECCO SSH", + "section": "", + "text": "Many of NASA’s current and legacy data collections are archive in netCDF4 format. By itself, netCDF4 are not cloud optimized and reading these files can take as long from a personal/local work environment as it takes to read the data from a working environment deployed in the cloud. Using Kerchunk, we can treat these files as cloud optimized assets by creating metadata json file describing existing netCDF4 files, their chunks, and where to access them. The json reference files can be read in using Zarr and Xarray for efficient reads and fast processing." + }, + { + "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#requirements", + "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#requirements", + "title": "PO.DAAC ECCO SSH", + "section": "Requirements", + "text": "Requirements\n\n1. AWS instance running in us-west-2\nNASA Earthdata Cloud data in S3 can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region.\n\n\n2. Earthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\n3. netrc File\nYou will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. For additional information see: Authentication for NASA Earthdata." + }, + { + "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#import-required-packages", + "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#import-required-packages", + "title": "PO.DAAC ECCO SSH", + "section": "Import required packages", + "text": "Import required packages\n\nimport requests\nimport xarray as xr\nimport ujson\nimport s3fs\nimport fsspec\nfrom tqdm import tqdm\nfrom glob import glob\nimport os\nimport pathlib\nimport hvplot.xarray\n\nfrom kerchunk.hdf import SingleHdf5ToZarr\nfrom kerchunk.combine import MultiZarrToZarr\n\n# The xarray produced from the reference file throws a SerializationWarning for each variable. Will need to explore why\nimport warnings\nwarnings.simplefilter(\"ignore\")" + }, + { + "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#create-dask-client-to-process-the-output-json-file-in-parallel", + "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#create-dask-client-to-process-the-output-json-file-in-parallel", + "title": "PO.DAAC ECCO SSH", + "section": "Create Dask client to process the output json file in parallel", + "text": "Create Dask client to process the output json file in parallel\nGenerating the Kerchunk reference file can take some time depending on the internal structure of the data. Dask allows us to execute the reference file generation process in parallel, thus speeding up the overall process.\n\nimport dask\nfrom dask.distributed import Client\nclient = Client(n_workers=4)\nclient\n\n2022-05-11 15:27:29,674 - distributed.diskutils - INFO - Found stale lock file and directory '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/dask-worker-space/worker-mezhdsy7', purging\n/srv/conda/envs/notebook/lib/python3.9/contextlib.py:126: UserWarning: Creating scratch directories is taking a surprisingly long time. This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.\n next(self.gen)\n\n\n\n \n \n Client\n Client-ddf55e52-d13e-11ec-818c-b6609e8b92a4\n \n\n\n\nConnection method: Cluster object\nCluster type: distributed.LocalCluster\n\n\nDashboard: http://127.0.0.1:41805/status\n\n\n\n\n\nCluster Info\n\n\n\n\n\n\nLocalCluster\na24e60d3\n\n\n\nDashboard: http://127.0.0.1:41805/status\nWorkers: 4\n\n\nTotal threads: 4\nTotal memory: 15.18 GiB\n\n\nStatus: running\nUsing processes: True\n\n\n\n\n\nScheduler Info\n\n\n\n\n\n\nScheduler\nScheduler-8e045442-a409-4c3b-8c8c-a95470883931\n\n\n\nComm: tcp://127.0.0.1:36901\nWorkers: 4\n\n\nDashboard: http://127.0.0.1:41805/status\nTotal threads: 4\n\n\nStarted: Just now\nTotal memory: 15.18 GiB\n\n\n\n\n\n\nWorkers\n\n\n\n\n\nWorker: 0\n\n\n\nComm: tcp://127.0.0.1:34235\nTotal threads: 1\n\n\nDashboard: http://127.0.0.1:42845/status\nMemory: 3.80 GiB\n\n\nNanny: tcp://127.0.0.1:37927\n\n\n\nLocal directory: /home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/dask-worker-space/worker-869qv5xb\n\n\n\n\n\n\n\n\n\n\nWorker: 1\n\n\n\nComm: tcp://127.0.0.1:40997\nTotal threads: 1\n\n\nDashboard: http://127.0.0.1:41189/status\nMemory: 3.80 GiB\n\n\nNanny: tcp://127.0.0.1:35257\n\n\n\nLocal directory: /home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/dask-worker-space/worker-3mo0d80c\n\n\n\n\n\n\n\n\n\n\nWorker: 2\n\n\n\nComm: tcp://127.0.0.1:46429\nTotal threads: 1\n\n\nDashboard: http://127.0.0.1:42211/status\nMemory: 3.80 GiB\n\n\nNanny: tcp://127.0.0.1:34287\n\n\n\nLocal directory: /home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/dask-worker-space/worker-o2fvmao4\n\n\n\n\n\n\n\n\n\n\nWorker: 3\n\n\n\nComm: tcp://127.0.0.1:41615\nTotal threads: 1\n\n\nDashboard: http://127.0.0.1:41507/status\nMemory: 3.80 GiB\n\n\nNanny: tcp://127.0.0.1:43053\n\n\n\nLocal directory: /home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/dask-worker-space/worker-9u77hywd" + }, + { + "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#get-temporary-s3-credentials", + "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#get-temporary-s3-credentials", + "title": "PO.DAAC ECCO SSH", + "section": "Get temporary S3 credentials", + "text": "Get temporary S3 credentials\nTemporary S3 credentials need to be passed to AWS. Note, these credentials must be refreshed after 1 hour.\n\ns3_cred_endpoint = {\n 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n 'ornldaac':'https://data.ornldaac.earthdata.nasa.gov/s3credentials',\n 'gesdisc':'https://data.gesdisc.earthdata.nasa.gov/s3credentials'\n}\n\n\ndef get_temp_creds():\n temp_creds_url = s3_cred_endpoint['podaac']\n return requests.get(temp_creds_url).json()\n\n\ntemp_creds_req = get_temp_creds()" + }, + { + "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#direct-access-a-single-netcdf4-file", + "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#direct-access-a-single-netcdf4-file", + "title": "PO.DAAC ECCO SSH", + "section": "Direct Access a single netCDF4 file", + "text": "Direct Access a single netCDF4 file\nPass temporary credentials to our filesystem object to access the S3 assets\n\nfs = s3fs.S3FileSystem(\n anon=False,\n key=temp_creds_req['accessKeyId'],\n secret=temp_creds_req['secretAccessKey'],\n token=temp_creds_req['sessionToken']\n)\n\n\nurl = 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-01_ECCO_V4r4_latlon_0p50deg.nc'\n\n\ns3_file_obj = fs.open(url, mode='rb')\n\nTime how long it takes to directly access a cloud asset for comparisons later.\n\n%%time\n\nxr_ds = xr.open_dataset(s3_file_obj, chunks='auto', engine='h5netcdf')\nxr_ds\n\nCPU times: user 228 ms, sys: 8.51 ms, total: 237 ms\nWall time: 272 ms\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (time: 1, latitude: 360, longitude: 720, nv: 2)\nCoordinates:\n * time (time) datetime64[ns] 2015-01-16T12:00:00\n * latitude (latitude) float32 -89.75 -89.25 -88.75 ... 89.25 89.75\n * longitude (longitude) float32 -179.8 -179.2 -178.8 ... 179.2 179.8\n time_bnds (time, nv) datetime64[ns] dask.array<chunksize=(1, 2), meta=np.ndarray>\n latitude_bnds (latitude, nv) float32 dask.array<chunksize=(360, 2), meta=np.ndarray>\n longitude_bnds (longitude, nv) float32 dask.array<chunksize=(720, 2), meta=np.ndarray>\nDimensions without coordinates: nv\nData variables:\n SSH (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n SSHIBC (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n SSHNOIBC (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\nAttributes: (12/57)\n acknowledgement: This research was carried out by the Jet Pr...\n author: Ian Fenty and Ou Wang\n cdm_data_type: Grid\n comment: Fields provided on a regular lat-lon grid. ...\n Conventions: CF-1.8, ACDD-1.3\n coordinates_comment: Note: the global 'coordinates' attribute de...\n ... ...\n time_coverage_duration: P1M\n time_coverage_end: 2015-02-01T00:00:00\n time_coverage_resolution: P1M\n time_coverage_start: 2015-01-01T00:00:00\n title: ECCO Sea Surface Height - Monthly Mean 0.5 ...\n uuid: 088d03b8-4158-11eb-876b-0cc47a3f47f1xarray.DatasetDimensions:time: 1latitude: 360longitude: 720nv: 2Coordinates: (6)time(time)datetime64[ns]2015-01-16T12:00:00axis :Tbounds :time_bndscoverage_content_type :coordinatelong_name :center time of averaging periodstandard_name :timearray(['2015-01-16T12:00:00.000000000'], dtype='datetime64[ns]')latitude(latitude)float32-89.75 -89.25 ... 89.25 89.75axis :Ybounds :latitude_bndscomment :uniform grid spacing from -89.75 to 89.75 by 0.5coverage_content_type :coordinatelong_name :latitude at grid cell centerstandard_name :latitudeunits :degrees_northarray([-89.75, -89.25, -88.75, ..., 88.75, 89.25, 89.75], dtype=float32)longitude(longitude)float32-179.8 -179.2 ... 179.2 179.8axis :Xbounds :longitude_bndscomment :uniform grid spacing from -179.75 to 179.75 by 0.5coverage_content_type :coordinatelong_name :longitude at grid cell centerstandard_name :longitudeunits :degrees_eastarray([-179.75, -179.25, -178.75, ..., 178.75, 179.25, 179.75],\n dtype=float32)time_bnds(time, nv)datetime64[ns]dask.array<chunksize=(1, 2), meta=np.ndarray>comment :Start and end times of averaging period.coverage_content_type :coordinatelong_name :time bounds of averaging period\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n16 B\n16 B\n\n\nShape\n(1, 2)\n(1, 2)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\ndatetime64[ns]\nnumpy.ndarray\n\n\n\n\n\n\n\n\nlatitude_bnds\n\n\n(latitude, nv)\n\n\nfloat32\n\n\ndask.array<chunksize=(360, 2), meta=np.ndarray>\n\n\n\n\ncoverage_content_type :\n\ncoordinate\n\nlong_name :\n\nlatitude bounds grid cells\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n2.81 kiB\n2.81 kiB\n\n\nShape\n(360, 2)\n(360, 2)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nlongitude_bnds\n\n\n(longitude, nv)\n\n\nfloat32\n\n\ndask.array<chunksize=(720, 2), meta=np.ndarray>\n\n\n\n\ncoverage_content_type :\n\ncoordinate\n\nlong_name :\n\nlongitude bounds grid cells\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n5.62 kiB\n5.62 kiB\n\n\nShape\n(720, 2)\n(720, 2)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nData variables: (3)\n\n\n\n\n\nSSH\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nDynamic sea surface height anomaly\n\nstandard_name :\n\nsea_surface_height_above_geoid\n\nunits :\n\nm\n\ncomment :\n\nDynamic sea surface height anomaly above the geoid, suitable for comparisons with altimetry sea surface height data products that apply the inverse barometer (IB) correction. Note: SSH is calculated by correcting model sea level anomaly ETAN for three effects: a) global mean steric sea level changes related to density changes in the Boussinesq volume-conserving model (Greatbatch correction, see sterGloH), b) the inverted barometer (IB) effect (see SSHIBC) and c) sea level displacement due to sea-ice and snow pressure loading (see sIceLoad). SSH can be compared with the similarly-named SSH variable in previous ECCO products that did not include atmospheric pressure loading (e.g., Version 4 Release 3). Use SSHNOIBC for comparisons with altimetry data products that do NOT apply the IB correction.\n\nvalid_min :\n\n-1.8805772066116333\n\nvalid_max :\n\n1.4207719564437866\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.99 MiB\n0.99 MiB\n\n\nShape\n(1, 360, 720)\n(1, 360, 720)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSSHIBC\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nThe inverted barometer (IB) correction to sea surface height due to atmospheric pressure loading\n\nunits :\n\nm\n\ncomment :\n\nNot an SSH itself, but a correction to model sea level anomaly (ETAN) required to account for the static part of sea surface displacement by atmosphere pressure loading: SSH = SSHNOIBC - SSHIBC. Note: Use SSH for model-data comparisons with altimetry data products that DO apply the IB correction and SSHNOIBC for comparisons with altimetry data products that do NOT apply the IB correction.\n\nvalid_min :\n\n-0.30144819617271423\n\nvalid_max :\n\n0.5245633721351624\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.99 MiB\n0.99 MiB\n\n\nShape\n(1, 360, 720)\n(1, 360, 720)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSSHNOIBC\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nSea surface height anomaly without the inverted barometer (IB) correction\n\nunits :\n\nm\n\ncomment :\n\nSea surface height anomaly above the geoid without the inverse barometer (IB) correction, suitable for comparisons with altimetry sea surface height data products that do NOT apply the inverse barometer (IB) correction. Note: SSHNOIBC is calculated by correcting model sea level anomaly ETAN for two effects: a) global mean steric sea level changes related to density changes in the Boussinesq volume-conserving model (Greatbatch correction, see sterGloH), b) sea level displacement due to sea-ice and snow pressure loading (see sIceLoad). In ECCO Version 4 Release 4 the model is forced with atmospheric pressure loading. SSHNOIBC does not correct for the static part of the effect of atmosphere pressure loading on sea surface height (the so-called inverse barometer (IB) correction). Use SSH for comparisons with altimetry data products that DO apply the IB correction.\n\nvalid_min :\n\n-1.6654272079467773\n\nvalid_max :\n\n1.4550364017486572\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.99 MiB\n0.99 MiB\n\n\nShape\n(1, 360, 720)\n(1, 360, 720)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nAttributes: (57)acknowledgement :This research was carried out by the Jet Propulsion Laboratory, managed by the California Institute of Technology under a contract with the National Aeronautics and Space Administration.author :Ian Fenty and Ou Wangcdm_data_type :Gridcomment :Fields provided on a regular lat-lon grid. They have been mapped to the regular lat-lon grid from the original ECCO lat-lon-cap 90 (llc90) native model grid. SSH (dynamic sea surface height) = SSHNOIBC (dynamic sea surface without the inverse barometer correction) - SSHIBC (inverse barometer correction). The inverted barometer correction accounts for variations in sea surface height due to atmospheric pressure variations.Conventions :CF-1.8, ACDD-1.3coordinates_comment :Note: the global 'coordinates' attribute describes auxillary coordinates.creator_email :ecco-group@mit.educreator_institution :NASA Jet Propulsion Laboratory (JPL)creator_name :ECCO Consortiumcreator_type :groupcreator_url :https://ecco-group.orgdate_created :2020-12-18T09:39:51date_issued :2020-12-18T09:39:51date_metadata_modified :2021-03-15T22:07:49date_modified :2021-03-15T22:07:49geospatial_bounds_crs :EPSG:4326geospatial_lat_max :90.0geospatial_lat_min :-90.0geospatial_lat_resolution :0.5geospatial_lat_units :degrees_northgeospatial_lon_max :180.0geospatial_lon_min :-180.0geospatial_lon_resolution :0.5geospatial_lon_units :degrees_easthistory :Inaugural release of an ECCO Central Estimate solution to PO.DAACid :10.5067/ECG5M-SSH44institution :NASA Jet Propulsion Laboratory (JPL)instrument_vocabulary :GCMD instrument keywordskeywords :EARTH SCIENCE > OCEANS > SEA SURFACE TOPOGRAPHY > SEA SURFACE HEIGHT, EARTH SCIENCE SERVICES > MODELS > EARTH SCIENCE REANALYSES/ASSIMILATION MODELSkeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordslicense :Public Domainmetadata_link :https://cmr.earthdata.nasa.gov/search/collections.umm_json?ShortName=ECCO_L4_SSH_05DEG_MONTHLY_V4R4naming_authority :gov.nasa.jplplatform :ERS-1/2, TOPEX/Poseidon, Geosat Follow-On (GFO), ENVISAT, Jason-1, Jason-2, CryoSat-2, SARAL/AltiKa, Jason-3, AVHRR, Aquarius, SSM/I, SSMIS, GRACE, DTU17MDT, Argo, WOCE, GO-SHIP, MEOP, Ice Tethered Profilers (ITP)platform_vocabulary :GCMD platform keywordsprocessing_level :L4product_name :SEA_SURFACE_HEIGHT_mon_mean_2015-01_ECCO_V4r4_latlon_0p50deg.ncproduct_time_coverage_end :2018-01-01T00:00:00product_time_coverage_start :1992-01-01T12:00:00product_version :Version 4, Release 4program :NASA Physical Oceanography, Cryosphere, Modeling, Analysis, and Prediction (MAP)project :Estimating the Circulation and Climate of the Ocean (ECCO)publisher_email :podaac@podaac.jpl.nasa.govpublisher_institution :PO.DAACpublisher_name :Physical Oceanography Distributed Active Archive Center (PO.DAAC)publisher_type :institutionpublisher_url :https://podaac.jpl.nasa.govreferences :ECCO Consortium, Fukumori, I., Wang, O., Fenty, I., Forget, G., Heimbach, P., & Ponte, R. M. 2020. Synopsis of the ECCO Central Production Global Ocean and Sea-Ice State Estimate (Version 4 Release 4). doi:10.5281/zenodo.3765928source :The ECCO V4r4 state estimate was produced by fitting a free-running solution of the MITgcm (checkpoint 66g) to satellite and in situ observational data in a least squares sense using the adjoint methodstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventionsummary :This dataset provides monthly-averaged dynamic sea surface height interpolated to a regular 0.5-degree grid from the ECCO Version 4 Release 4 (V4r4) ocean and sea-ice state estimate. Estimating the Circulation and Climate of the Ocean (ECCO) state estimates are dynamically and kinematically-consistent reconstructions of the three-dimensional, time-evolving ocean, sea-ice, and surface atmospheric states. ECCO V4r4 is a free-running solution of a global, nominally 1-degree configuration of the MIT general circulation model (MITgcm) that has been fit to observations in a least-squares sense. Observational data constraints used in V4r4 include sea surface height (SSH) from satellite altimeters [ERS-1/2, TOPEX/Poseidon, GFO, ENVISAT, Jason-1,2,3, CryoSat-2, and SARAL/AltiKa]; sea surface temperature (SST) from satellite radiometers [AVHRR], sea surface salinity (SSS) from the Aquarius satellite radiometer/scatterometer, ocean bottom pressure (OBP) from the GRACE satellite gravimeter; sea-ice concentration from satellite radiometers [SSM/I and SSMIS], and in-situ ocean temperature and salinity measured with conductivity-temperature-depth (CTD) sensors and expendable bathythermographs (XBTs) from several programs [e.g., WOCE, GO-SHIP, Argo, and others] and platforms [e.g., research vessels, gliders, moorings, ice-tethered profilers, and instrumented pinnipeds]. V4r4 covers the period 1992-01-01T12:00:00 to 2018-01-01T00:00:00.time_coverage_duration :P1Mtime_coverage_end :2015-02-01T00:00:00time_coverage_resolution :P1Mtime_coverage_start :2015-01-01T00:00:00title :ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4)uuid :088d03b8-4158-11eb-876b-0cc47a3f47f1" + }, + { + "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#specify-a-list-of-s3-urls", + "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#specify-a-list-of-s3-urls", + "title": "PO.DAAC ECCO SSH", + "section": "Specify a list of S3 URLs", + "text": "Specify a list of S3 URLs\nData Collection: ECCO_L4_SSH_05DEG_MONTHLY_V4R4\nTime Range: 2015\n\nurls = ['s3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2014-12_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-01_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-02_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-03_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-04_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-05_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-06_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-07_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-08_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-09_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-10_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-11_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-12_ECCO_V4r4_latlon_0p50deg.nc']" + }, + { + "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#generate-the-kerchunk-reference-files.", + "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#generate-the-kerchunk-reference-files.", + "title": "PO.DAAC ECCO SSH", + "section": "Generate the Kerchunk reference files.", + "text": "Generate the Kerchunk reference files.\nDefine a function to generate the Kerchunk reference files. These files can take a little time to generate.\n\ndef gen_json(u):\n so = dict(\n mode= \"rb\", \n anon= False, \n default_fill_cache= False,\n default_cache_type= \"none\"\n )\n with fs.open(u, **so) as infile:\n h5chunks = SingleHdf5ToZarr(infile, u, inline_threshold=300)\n with open(f\"jsons/{u.split('/')[-1]}.json\", 'wb') as outf:\n outf.write(ujson.dumps(h5chunks.translate()).encode())\n\nCreate output jsons directory if one does not exist.\n\npathlib.Path('./jsons/').mkdir(exist_ok=True)\n\nUse the Dask Delayed function to create the Kerchunk reference file for each URL from the list of URLs in parallel\n\n%%time\n\nreference_files = []\nfor url in urls:\n ref = dask.delayed(gen_json)(url)\n reference_files.append(ref)\n\nreference_files_compute = dask.compute(*reference_files)\n\nCPU times: user 195 ms, sys: 83.4 ms, total: 278 ms\nWall time: 1.38 s\n\n\n\nfs_ref_list = fsspec.filesystem('file')\n\n\nreference_list = sorted([x for x in fs_ref_list.ls('jsons') if '.json' in x])\nreference_list\n\n['/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2014-12_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-01_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-02_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-03_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-04_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-05_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-06_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-07_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-08_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-09_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-10_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-11_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-12_ECCO_V4r4_latlon_0p50deg.nc.json']" + }, + { + "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#read-single-netcdf4-using-kerchunk-reference-file", + "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#read-single-netcdf4-using-kerchunk-reference-file", + "title": "PO.DAAC ECCO SSH", + "section": "Read single netCDF4 using Kerchunk reference file", + "text": "Read single netCDF4 using Kerchunk reference file\nOpen the first reference file to read into an xarray dataset\n\nwith open(reference_list[0]) as j:\n reference = ujson.load(j)\n\nSet configurations options\n\ns_opts = {'skip_instance_cache':True} #json\nr_opts = {'anon':False, \n 'key':temp_creds_req['accessKeyId'], \n 'secret':temp_creds_req['secretAccessKey'], \n 'token':temp_creds_req['sessionToken']} #ncfiles\n\n\nfs_single = fsspec.filesystem(\"reference\",\n fo=reference,\n ref_storage_args=s_opts,\n remote_protocol='s3', \n remote_options=r_opts)\n\nRead in a single reference object. We get a lot of SerializationWarnings which are ignored here using the warning package.\nNOTE, the fill value, data range, min value, and max value may not match the source file. Will need to look into this more.\n\n%%time\n\nm = fs_single.get_mapper(\"\")\nds_single = xr.open_dataset(m, engine=\"zarr\", backend_kwargs={'consolidated':False}, chunks={})\nds_single\n\nCPU times: user 56.3 ms, sys: 26 ms, total: 82.2 ms\nWall time: 221 ms\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (time: 1, latitude: 360, longitude: 720, nv: 2)\nCoordinates:\n * latitude (latitude) float32 -89.75 -89.25 -88.75 ... 89.25 89.75\n latitude_bnds (latitude, nv) float32 dask.array<chunksize=(360, 2), meta=np.ndarray>\n * longitude (longitude) float32 -179.8 -179.2 -178.8 ... 179.2 179.8\n longitude_bnds (longitude, nv) float32 dask.array<chunksize=(720, 2), meta=np.ndarray>\n * time (time) datetime64[ns] 2014-12-16T12:00:00\n time_bnds (time, nv) datetime64[ns] dask.array<chunksize=(1, 2), meta=np.ndarray>\nDimensions without coordinates: nv\nData variables:\n SSH (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n SSHIBC (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n SSHNOIBC (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\nAttributes: (12/57)\n Conventions: CF-1.8, ACDD-1.3\n acknowledgement: This research was carried out by the Jet Pr...\n author: Ian Fenty and Ou Wang\n cdm_data_type: Grid\n comment: Fields provided on a regular lat-lon grid. ...\n coordinates_comment: Note: the global 'coordinates' attribute de...\n ... ...\n time_coverage_duration: P1M\n time_coverage_end: 2015-01-01T00:00:00\n time_coverage_resolution: P1M\n time_coverage_start: 2014-12-01T00:00:00\n title: ECCO Sea Surface Height - Monthly Mean 0.5 ...\n uuid: 08a2fc68-4158-11eb-b498-0cc47a3f6943xarray.DatasetDimensions:time: 1latitude: 360longitude: 720nv: 2Coordinates: (6)latitude(latitude)float32-89.75 -89.25 ... 89.25 89.75axis :Ybounds :latitude_bndscomment :uniform grid spacing from -89.75 to 89.75 by 0.5coverage_content_type :coordinatelong_name :latitude at grid cell centerstandard_name :latitudeunits :degrees_northarray([-89.75, -89.25, -88.75, ..., 88.75, 89.25, 89.75], dtype=float32)latitude_bnds(latitude, nv)float32dask.array<chunksize=(360, 2), meta=np.ndarray>coverage_content_type :coordinatelong_name :latitude bounds grid cells\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n2.81 kiB\n2.81 kiB\n\n\nShape\n(360, 2)\n(360, 2)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nlongitude\n\n\n(longitude)\n\n\nfloat32\n\n\n-179.8 -179.2 ... 179.2 179.8\n\n\n\n\naxis :\n\nX\n\nbounds :\n\nlongitude_bnds\n\ncomment :\n\nuniform grid spacing from -179.75 to 179.75 by 0.5\n\ncoverage_content_type :\n\ncoordinate\n\nlong_name :\n\nlongitude at grid cell center\n\nstandard_name :\n\nlongitude\n\nunits :\n\ndegrees_east\n\n\n\n\narray([-179.75, -179.25, -178.75, ..., 178.75, 179.25, 179.75],\n dtype=float32)\n\n\nlongitude_bnds\n\n\n(longitude, nv)\n\n\nfloat32\n\n\ndask.array<chunksize=(720, 2), meta=np.ndarray>\n\n\n\n\ncoverage_content_type :\n\ncoordinate\n\nlong_name :\n\nlongitude bounds grid cells\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n5.62 kiB\n5.62 kiB\n\n\nShape\n(720, 2)\n(720, 2)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\ntime\n\n\n(time)\n\n\ndatetime64[ns]\n\n\n2014-12-16T12:00:00\n\n\n\n\naxis :\n\nT\n\nbounds :\n\ntime_bnds\n\ncoverage_content_type :\n\ncoordinate\n\nlong_name :\n\ncenter time of averaging period\n\nstandard_name :\n\ntime\n\n\n\n\narray(['2014-12-16T12:00:00.000000000'], dtype='datetime64[ns]')\n\n\ntime_bnds\n\n\n(time, nv)\n\n\ndatetime64[ns]\n\n\ndask.array<chunksize=(1, 2), meta=np.ndarray>\n\n\n\n\ncomment :\n\nStart and end times of averaging period.\n\ncoverage_content_type :\n\ncoordinate\n\nlong_name :\n\ntime bounds of averaging period\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n16 B\n16 B\n\n\nShape\n(1, 2)\n(1, 2)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\ndatetime64[ns]\nnumpy.ndarray\n\n\n\n\n\n\n\n\nData variables: (3)\n\n\n\n\n\nSSH\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncomment :\n\nDynamic sea surface height anomaly above the geoid, suitable for comparisons with altimetry sea surface height data products that apply the inverse barometer (IB) correction. Note: SSH is calculated by correcting model sea level anomaly ETAN for three effects: a) global mean steric sea level changes related to density changes in the Boussinesq volume-conserving model (Greatbatch correction, see sterGloH), b) the inverted barometer (IB) effect (see SSHIBC) and c) sea level displacement due to sea-ice and snow pressure loading (see sIceLoad). SSH can be compared with the similarly-named SSH variable in previous ECCO products that did not include atmospheric pressure loading (e.g., Version 4 Release 3). Use SSHNOIBC for comparisons with altimetry data products that do NOT apply the IB correction.\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nDynamic sea surface height anomaly\n\nstandard_name :\n\nsea_surface_height_above_geoid\n\nunits :\n\nm\n\nvalid_max :\n\n1.4207719564437866\n\nvalid_min :\n\n-1.8805772066116333\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.99 MiB\n0.99 MiB\n\n\nShape\n(1, 360, 720)\n(1, 360, 720)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSSHIBC\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncomment :\n\nNot an SSH itself, but a correction to model sea level anomaly (ETAN) required to account for the static part of sea surface displacement by atmosphere pressure loading: SSH = SSHNOIBC - SSHIBC. Note: Use SSH for model-data comparisons with altimetry data products that DO apply the IB correction and SSHNOIBC for comparisons with altimetry data products that do NOT apply the IB correction.\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nThe inverted barometer (IB) correction to sea surface height due to atmospheric pressure loading\n\nunits :\n\nm\n\nvalid_max :\n\n0.5245633721351624\n\nvalid_min :\n\n-0.30144819617271423\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.99 MiB\n0.99 MiB\n\n\nShape\n(1, 360, 720)\n(1, 360, 720)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSSHNOIBC\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncomment :\n\nSea surface height anomaly above the geoid without the inverse barometer (IB) correction, suitable for comparisons with altimetry sea surface height data products that do NOT apply the inverse barometer (IB) correction. Note: SSHNOIBC is calculated by correcting model sea level anomaly ETAN for two effects: a) global mean steric sea level changes related to density changes in the Boussinesq volume-conserving model (Greatbatch correction, see sterGloH), b) sea level displacement due to sea-ice and snow pressure loading (see sIceLoad). In ECCO Version 4 Release 4 the model is forced with atmospheric pressure loading. SSHNOIBC does not correct for the static part of the effect of atmosphere pressure loading on sea surface height (the so-called inverse barometer (IB) correction). Use SSH for comparisons with altimetry data products that DO apply the IB correction.\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nSea surface height anomaly without the inverted barometer (IB) correction\n\nunits :\n\nm\n\nvalid_max :\n\n1.4550364017486572\n\nvalid_min :\n\n-1.6654272079467773\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.99 MiB\n0.99 MiB\n\n\nShape\n(1, 360, 720)\n(1, 360, 720)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nAttributes: (57)Conventions :CF-1.8, ACDD-1.3acknowledgement :This research was carried out by the Jet Propulsion Laboratory, managed by the California Institute of Technology under a contract with the National Aeronautics and Space Administration.author :Ian Fenty and Ou Wangcdm_data_type :Gridcomment :Fields provided on a regular lat-lon grid. They have been mapped to the regular lat-lon grid from the original ECCO lat-lon-cap 90 (llc90) native model grid. SSH (dynamic sea surface height) = SSHNOIBC (dynamic sea surface without the inverse barometer correction) - SSHIBC (inverse barometer correction). The inverted barometer correction accounts for variations in sea surface height due to atmospheric pressure variations.coordinates_comment :Note: the global 'coordinates' attribute describes auxillary coordinates.creator_email :ecco-group@mit.educreator_institution :NASA Jet Propulsion Laboratory (JPL)creator_name :ECCO Consortiumcreator_type :groupcreator_url :https://ecco-group.orgdate_created :2020-12-18T09:39:51date_issued :2020-12-18T09:39:51date_metadata_modified :2021-03-15T22:07:49date_modified :2021-03-15T22:07:49geospatial_bounds_crs :EPSG:4326geospatial_lat_max :90.0geospatial_lat_min :-90.0geospatial_lat_resolution :0.5geospatial_lat_units :degrees_northgeospatial_lon_max :180.0geospatial_lon_min :-180.0geospatial_lon_resolution :0.5geospatial_lon_units :degrees_easthistory :Inaugural release of an ECCO Central Estimate solution to PO.DAACid :10.5067/ECG5M-SSH44institution :NASA Jet Propulsion Laboratory (JPL)instrument_vocabulary :GCMD instrument keywordskeywords :EARTH SCIENCE > OCEANS > SEA SURFACE TOPOGRAPHY > SEA SURFACE HEIGHT, EARTH SCIENCE SERVICES > MODELS > EARTH SCIENCE REANALYSES/ASSIMILATION MODELSkeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordslicense :Public Domainmetadata_link :https://cmr.earthdata.nasa.gov/search/collections.umm_json?ShortName=ECCO_L4_SSH_05DEG_MONTHLY_V4R4naming_authority :gov.nasa.jplplatform :ERS-1/2, TOPEX/Poseidon, Geosat Follow-On (GFO), ENVISAT, Jason-1, Jason-2, CryoSat-2, SARAL/AltiKa, Jason-3, AVHRR, Aquarius, SSM/I, SSMIS, GRACE, DTU17MDT, Argo, WOCE, GO-SHIP, MEOP, Ice Tethered Profilers (ITP)platform_vocabulary :GCMD platform keywordsprocessing_level :L4product_name :SEA_SURFACE_HEIGHT_mon_mean_2014-12_ECCO_V4r4_latlon_0p50deg.ncproduct_time_coverage_end :2018-01-01T00:00:00product_time_coverage_start :1992-01-01T12:00:00product_version :Version 4, Release 4program :NASA Physical Oceanography, Cryosphere, Modeling, Analysis, and Prediction (MAP)project :Estimating the Circulation and Climate of the Ocean (ECCO)publisher_email :podaac@podaac.jpl.nasa.govpublisher_institution :PO.DAACpublisher_name :Physical Oceanography Distributed Active Archive Center (PO.DAAC)publisher_type :institutionpublisher_url :https://podaac.jpl.nasa.govreferences :ECCO Consortium, Fukumori, I., Wang, O., Fenty, I., Forget, G., Heimbach, P., & Ponte, R. M. 2020. Synopsis of the ECCO Central Production Global Ocean and Sea-Ice State Estimate (Version 4 Release 4). doi:10.5281/zenodo.3765928source :The ECCO V4r4 state estimate was produced by fitting a free-running solution of the MITgcm (checkpoint 66g) to satellite and in situ observational data in a least squares sense using the adjoint methodstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventionsummary :This dataset provides monthly-averaged dynamic sea surface height interpolated to a regular 0.5-degree grid from the ECCO Version 4 Release 4 (V4r4) ocean and sea-ice state estimate. Estimating the Circulation and Climate of the Ocean (ECCO) state estimates are dynamically and kinematically-consistent reconstructions of the three-dimensional, time-evolving ocean, sea-ice, and surface atmospheric states. ECCO V4r4 is a free-running solution of a global, nominally 1-degree configuration of the MIT general circulation model (MITgcm) that has been fit to observations in a least-squares sense. Observational data constraints used in V4r4 include sea surface height (SSH) from satellite altimeters [ERS-1/2, TOPEX/Poseidon, GFO, ENVISAT, Jason-1,2,3, CryoSat-2, and SARAL/AltiKa]; sea surface temperature (SST) from satellite radiometers [AVHRR], sea surface salinity (SSS) from the Aquarius satellite radiometer/scatterometer, ocean bottom pressure (OBP) from the GRACE satellite gravimeter; sea-ice concentration from satellite radiometers [SSM/I and SSMIS], and in-situ ocean temperature and salinity measured with conductivity-temperature-depth (CTD) sensors and expendable bathythermographs (XBTs) from several programs [e.g., WOCE, GO-SHIP, Argo, and others] and platforms [e.g., research vessels, gliders, moorings, ice-tethered profilers, and instrumented pinnipeds]. V4r4 covers the period 1992-01-01T12:00:00 to 2018-01-01T00:00:00.time_coverage_duration :P1Mtime_coverage_end :2015-01-01T00:00:00time_coverage_resolution :P1Mtime_coverage_start :2014-12-01T00:00:00title :ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4)uuid :08a2fc68-4158-11eb-b498-0cc47a3f6943" + }, + { + "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#read-multiple-netcdf4-files-using-kerchunk-reference-file", + "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#read-multiple-netcdf4-files-using-kerchunk-reference-file", + "title": "PO.DAAC ECCO SSH", + "section": "Read multiple netCDF4 files using Kerchunk reference file", + "text": "Read multiple netCDF4 files using Kerchunk reference file\nCombine the individual reference files into a single time series reference object\n\n%%time\n\nds_k =[]\nfor ref in reference_list:\n s_opts = s_opts\n r_opts = r_opts\n fs = fsspec.filesystem(\"reference\",\n fo=ref,\n ref_storage_args=s_opts,\n remote_protocol='s3',\n remote_options=r_opts)\n m = fs.get_mapper(\"\")\n ds_k.append(xr.open_dataset(m, engine=\"zarr\", backend_kwargs={'consolidated':False}, chunks={}))\n \nds_multi = xr.concat(ds_k, dim='time')\n \nds_multi\n\nCPU times: user 735 ms, sys: 31.4 ms, total: 766 ms\nWall time: 3.57 s\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (time: 13, latitude: 360, longitude: 720, nv: 2)\nCoordinates:\n * latitude (latitude) float32 -89.75 -89.25 -88.75 ... 89.25 89.75\n latitude_bnds (latitude, nv) float32 -90.0 -89.5 -89.5 ... 89.5 89.5 90.0\n * longitude (longitude) float32 -179.8 -179.2 -178.8 ... 179.2 179.8\n longitude_bnds (longitude, nv) float32 -180.0 -179.5 -179.5 ... 179.5 180.0\n * time (time) datetime64[ns] 2014-12-16T12:00:00 ... 2015-12-16T...\n time_bnds (time, nv) datetime64[ns] dask.array<chunksize=(1, 2), meta=np.ndarray>\nDimensions without coordinates: nv\nData variables:\n SSH (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n SSHIBC (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n SSHNOIBC (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\nAttributes: (12/57)\n Conventions: CF-1.8, ACDD-1.3\n acknowledgement: This research was carried out by the Jet Pr...\n author: Ian Fenty and Ou Wang\n cdm_data_type: Grid\n comment: Fields provided on a regular lat-lon grid. ...\n coordinates_comment: Note: the global 'coordinates' attribute de...\n ... ...\n time_coverage_duration: P1M\n time_coverage_end: 2015-01-01T00:00:00\n time_coverage_resolution: P1M\n time_coverage_start: 2014-12-01T00:00:00\n title: ECCO Sea Surface Height - Monthly Mean 0.5 ...\n uuid: 08a2fc68-4158-11eb-b498-0cc47a3f6943xarray.DatasetDimensions:time: 13latitude: 360longitude: 720nv: 2Coordinates: (6)latitude(latitude)float32-89.75 -89.25 ... 89.25 89.75axis :Ybounds :latitude_bndscomment :uniform grid spacing from -89.75 to 89.75 by 0.5coverage_content_type :coordinatelong_name :latitude at grid cell centerstandard_name :latitudeunits :degrees_northarray([-89.75, -89.25, -88.75, ..., 88.75, 89.25, 89.75], dtype=float32)latitude_bnds(latitude, nv)float32-90.0 -89.5 -89.5 ... 89.5 90.0coverage_content_type :coordinatelong_name :latitude bounds grid cellsarray([[-90. , -89.5],\n [-89.5, -89. ],\n [-89. , -88.5],\n [-88.5, -88. ],\n [-88. , -87.5],\n [-87.5, -87. ],\n [-87. , -86.5],\n [-86.5, -86. ],\n [-86. , -85.5],\n [-85.5, -85. ],\n [-85. , -84.5],\n [-84.5, -84. ],\n [-84. , -83.5],\n [-83.5, -83. ],\n [-83. , -82.5],\n [-82.5, -82. ],\n [-82. , -81.5],\n [-81.5, -81. ],\n [-81. , -80.5],\n [-80.5, -80. ],\n...\n [ 80. , 80.5],\n [ 80.5, 81. ],\n [ 81. , 81.5],\n [ 81.5, 82. ],\n [ 82. , 82.5],\n [ 82.5, 83. ],\n [ 83. , 83.5],\n [ 83.5, 84. ],\n [ 84. , 84.5],\n [ 84.5, 85. ],\n [ 85. , 85.5],\n [ 85.5, 86. ],\n [ 86. , 86.5],\n [ 86.5, 87. ],\n [ 87. , 87.5],\n [ 87.5, 88. ],\n [ 88. , 88.5],\n [ 88.5, 89. ],\n [ 89. , 89.5],\n [ 89.5, 90. ]], dtype=float32)longitude(longitude)float32-179.8 -179.2 ... 179.2 179.8axis :Xbounds :longitude_bndscomment :uniform grid spacing from -179.75 to 179.75 by 0.5coverage_content_type :coordinatelong_name :longitude at grid cell centerstandard_name :longitudeunits :degrees_eastarray([-179.75, -179.25, -178.75, ..., 178.75, 179.25, 179.75],\n dtype=float32)longitude_bnds(longitude, nv)float32-180.0 -179.5 ... 179.5 180.0coverage_content_type :coordinatelong_name :longitude bounds grid cellsarray([[-180. , -179.5],\n [-179.5, -179. ],\n [-179. , -178.5],\n ...,\n [ 178.5, 179. ],\n [ 179. , 179.5],\n [ 179.5, 180. ]], dtype=float32)time(time)datetime64[ns]2014-12-16T12:00:00 ... 2015-12-...axis :Tbounds :time_bndscoverage_content_type :coordinatelong_name :center time of averaging periodstandard_name :timearray(['2014-12-16T12:00:00.000000000', '2015-01-16T12:00:00.000000000',\n '2015-02-15T00:00:00.000000000', '2015-03-16T12:00:00.000000000',\n '2015-04-16T00:00:00.000000000', '2015-05-16T12:00:00.000000000',\n '2015-06-16T00:00:00.000000000', '2015-07-16T12:00:00.000000000',\n '2015-08-16T12:00:00.000000000', '2015-09-16T00:00:00.000000000',\n '2015-10-16T12:00:00.000000000', '2015-11-16T00:00:00.000000000',\n '2015-12-16T12:00:00.000000000'], dtype='datetime64[ns]')time_bnds(time, nv)datetime64[ns]dask.array<chunksize=(1, 2), meta=np.ndarray>comment :Start and end times of averaging period.coverage_content_type :coordinatelong_name :time bounds of averaging period\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n208 B\n16 B\n\n\nShape\n(13, 2)\n(1, 2)\n\n\nCount\n39 Tasks\n13 Chunks\n\n\nType\ndatetime64[ns]\nnumpy.ndarray\n\n\n\n\n\n\n\nData variables: (3)\n\n\n\n\n\nSSH\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncomment :\n\nDynamic sea surface height anomaly above the geoid, suitable for comparisons with altimetry sea surface height data products that apply the inverse barometer (IB) correction. Note: SSH is calculated by correcting model sea level anomaly ETAN for three effects: a) global mean steric sea level changes related to density changes in the Boussinesq volume-conserving model (Greatbatch correction, see sterGloH), b) the inverted barometer (IB) effect (see SSHIBC) and c) sea level displacement due to sea-ice and snow pressure loading (see sIceLoad). SSH can be compared with the similarly-named SSH variable in previous ECCO products that did not include atmospheric pressure loading (e.g., Version 4 Release 3). Use SSHNOIBC for comparisons with altimetry data products that do NOT apply the IB correction.\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nDynamic sea surface height anomaly\n\nstandard_name :\n\nsea_surface_height_above_geoid\n\nunits :\n\nm\n\nvalid_max :\n\n1.4207719564437866\n\nvalid_min :\n\n-1.8805772066116333\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n12.85 MiB\n0.99 MiB\n\n\nShape\n(13, 360, 720)\n(1, 360, 720)\n\n\nCount\n39 Tasks\n13 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSSHIBC\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncomment :\n\nNot an SSH itself, but a correction to model sea level anomaly (ETAN) required to account for the static part of sea surface displacement by atmosphere pressure loading: SSH = SSHNOIBC - SSHIBC. Note: Use SSH for model-data comparisons with altimetry data products that DO apply the IB correction and SSHNOIBC for comparisons with altimetry data products that do NOT apply the IB correction.\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nThe inverted barometer (IB) correction to sea surface height due to atmospheric pressure loading\n\nunits :\n\nm\n\nvalid_max :\n\n0.5245633721351624\n\nvalid_min :\n\n-0.30144819617271423\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n12.85 MiB\n0.99 MiB\n\n\nShape\n(13, 360, 720)\n(1, 360, 720)\n\n\nCount\n39 Tasks\n13 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSSHNOIBC\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncomment :\n\nSea surface height anomaly above the geoid without the inverse barometer (IB) correction, suitable for comparisons with altimetry sea surface height data products that do NOT apply the inverse barometer (IB) correction. Note: SSHNOIBC is calculated by correcting model sea level anomaly ETAN for two effects: a) global mean steric sea level changes related to density changes in the Boussinesq volume-conserving model (Greatbatch correction, see sterGloH), b) sea level displacement due to sea-ice and snow pressure loading (see sIceLoad). In ECCO Version 4 Release 4 the model is forced with atmospheric pressure loading. SSHNOIBC does not correct for the static part of the effect of atmosphere pressure loading on sea surface height (the so-called inverse barometer (IB) correction). Use SSH for comparisons with altimetry data products that DO apply the IB correction.\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nSea surface height anomaly without the inverted barometer (IB) correction\n\nunits :\n\nm\n\nvalid_max :\n\n1.4550364017486572\n\nvalid_min :\n\n-1.6654272079467773\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n12.85 MiB\n0.99 MiB\n\n\nShape\n(13, 360, 720)\n(1, 360, 720)\n\n\nCount\n39 Tasks\n13 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nAttributes: (57)Conventions :CF-1.8, ACDD-1.3acknowledgement :This research was carried out by the Jet Propulsion Laboratory, managed by the California Institute of Technology under a contract with the National Aeronautics and Space Administration.author :Ian Fenty and Ou Wangcdm_data_type :Gridcomment :Fields provided on a regular lat-lon grid. They have been mapped to the regular lat-lon grid from the original ECCO lat-lon-cap 90 (llc90) native model grid. SSH (dynamic sea surface height) = SSHNOIBC (dynamic sea surface without the inverse barometer correction) - SSHIBC (inverse barometer correction). The inverted barometer correction accounts for variations in sea surface height due to atmospheric pressure variations.coordinates_comment :Note: the global 'coordinates' attribute describes auxillary coordinates.creator_email :ecco-group@mit.educreator_institution :NASA Jet Propulsion Laboratory (JPL)creator_name :ECCO Consortiumcreator_type :groupcreator_url :https://ecco-group.orgdate_created :2020-12-18T09:39:51date_issued :2020-12-18T09:39:51date_metadata_modified :2021-03-15T22:07:49date_modified :2021-03-15T22:07:49geospatial_bounds_crs :EPSG:4326geospatial_lat_max :90.0geospatial_lat_min :-90.0geospatial_lat_resolution :0.5geospatial_lat_units :degrees_northgeospatial_lon_max :180.0geospatial_lon_min :-180.0geospatial_lon_resolution :0.5geospatial_lon_units :degrees_easthistory :Inaugural release of an ECCO Central Estimate solution to PO.DAACid :10.5067/ECG5M-SSH44institution :NASA Jet Propulsion Laboratory (JPL)instrument_vocabulary :GCMD instrument keywordskeywords :EARTH SCIENCE > OCEANS > SEA SURFACE TOPOGRAPHY > SEA SURFACE HEIGHT, EARTH SCIENCE SERVICES > MODELS > EARTH SCIENCE REANALYSES/ASSIMILATION MODELSkeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordslicense :Public Domainmetadata_link :https://cmr.earthdata.nasa.gov/search/collections.umm_json?ShortName=ECCO_L4_SSH_05DEG_MONTHLY_V4R4naming_authority :gov.nasa.jplplatform :ERS-1/2, TOPEX/Poseidon, Geosat Follow-On (GFO), ENVISAT, Jason-1, Jason-2, CryoSat-2, SARAL/AltiKa, Jason-3, AVHRR, Aquarius, SSM/I, SSMIS, GRACE, DTU17MDT, Argo, WOCE, GO-SHIP, MEOP, Ice Tethered Profilers (ITP)platform_vocabulary :GCMD platform keywordsprocessing_level :L4product_name :SEA_SURFACE_HEIGHT_mon_mean_2014-12_ECCO_V4r4_latlon_0p50deg.ncproduct_time_coverage_end :2018-01-01T00:00:00product_time_coverage_start :1992-01-01T12:00:00product_version :Version 4, Release 4program :NASA Physical Oceanography, Cryosphere, Modeling, Analysis, and Prediction (MAP)project :Estimating the Circulation and Climate of the Ocean (ECCO)publisher_email :podaac@podaac.jpl.nasa.govpublisher_institution :PO.DAACpublisher_name :Physical Oceanography Distributed Active Archive Center (PO.DAAC)publisher_type :institutionpublisher_url :https://podaac.jpl.nasa.govreferences :ECCO Consortium, Fukumori, I., Wang, O., Fenty, I., Forget, G., Heimbach, P., & Ponte, R. M. 2020. Synopsis of the ECCO Central Production Global Ocean and Sea-Ice State Estimate (Version 4 Release 4). doi:10.5281/zenodo.3765928source :The ECCO V4r4 state estimate was produced by fitting a free-running solution of the MITgcm (checkpoint 66g) to satellite and in situ observational data in a least squares sense using the adjoint methodstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventionsummary :This dataset provides monthly-averaged dynamic sea surface height interpolated to a regular 0.5-degree grid from the ECCO Version 4 Release 4 (V4r4) ocean and sea-ice state estimate. Estimating the Circulation and Climate of the Ocean (ECCO) state estimates are dynamically and kinematically-consistent reconstructions of the three-dimensional, time-evolving ocean, sea-ice, and surface atmospheric states. ECCO V4r4 is a free-running solution of a global, nominally 1-degree configuration of the MIT general circulation model (MITgcm) that has been fit to observations in a least-squares sense. Observational data constraints used in V4r4 include sea surface height (SSH) from satellite altimeters [ERS-1/2, TOPEX/Poseidon, GFO, ENVISAT, Jason-1,2,3, CryoSat-2, and SARAL/AltiKa]; sea surface temperature (SST) from satellite radiometers [AVHRR], sea surface salinity (SSS) from the Aquarius satellite radiometer/scatterometer, ocean bottom pressure (OBP) from the GRACE satellite gravimeter; sea-ice concentration from satellite radiometers [SSM/I and SSMIS], and in-situ ocean temperature and salinity measured with conductivity-temperature-depth (CTD) sensors and expendable bathythermographs (XBTs) from several programs [e.g., WOCE, GO-SHIP, Argo, and others] and platforms [e.g., research vessels, gliders, moorings, ice-tethered profilers, and instrumented pinnipeds]. V4r4 covers the period 1992-01-01T12:00:00 to 2018-01-01T00:00:00.time_coverage_duration :P1Mtime_coverage_end :2015-01-01T00:00:00time_coverage_resolution :P1Mtime_coverage_start :2014-12-01T00:00:00title :ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4)uuid :08a2fc68-4158-11eb-b498-0cc47a3f6943\n\n\n\nds_multi['SSH']\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray 'SSH' (time: 13, latitude: 360, longitude: 720)>\ndask.array<concatenate, shape=(13, 360, 720), dtype=float32, chunksize=(1, 360, 720), chunktype=numpy.ndarray>\nCoordinates:\n * latitude (latitude) float32 -89.75 -89.25 -88.75 ... 88.75 89.25 89.75\n * longitude (longitude) float32 -179.8 -179.2 -178.8 ... 178.8 179.2 179.8\n * time (time) datetime64[ns] 2014-12-16T12:00:00 ... 2015-12-16T12:00:00\nAttributes:\n comment: Dynamic sea surface height anomaly above the geoi...\n coverage_content_type: modelResult\n long_name: Dynamic sea surface height anomaly\n standard_name: sea_surface_height_above_geoid\n units: m\n valid_max: 1.4207719564437866\n valid_min: -1.8805772066116333xarray.DataArray'SSH'time: 13latitude: 360longitude: 720dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n12.85 MiB\n0.99 MiB\n\n\nShape\n(13, 360, 720)\n(1, 360, 720)\n\n\nCount\n39 Tasks\n13 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nCoordinates: (3)latitude(latitude)float32-89.75 -89.25 ... 89.25 89.75axis :Ybounds :latitude_bndscomment :uniform grid spacing from -89.75 to 89.75 by 0.5coverage_content_type :coordinatelong_name :latitude at grid cell centerstandard_name :latitudeunits :degrees_northarray([-89.75, -89.25, -88.75, ..., 88.75, 89.25, 89.75], dtype=float32)longitude(longitude)float32-179.8 -179.2 ... 179.2 179.8axis :Xbounds :longitude_bndscomment :uniform grid spacing from -179.75 to 179.75 by 0.5coverage_content_type :coordinatelong_name :longitude at grid cell centerstandard_name :longitudeunits :degrees_eastarray([-179.75, -179.25, -178.75, ..., 178.75, 179.25, 179.75],\n dtype=float32)time(time)datetime64[ns]2014-12-16T12:00:00 ... 2015-12-...axis :Tbounds :time_bndscoverage_content_type :coordinatelong_name :center time of averaging periodstandard_name :timearray(['2014-12-16T12:00:00.000000000', '2015-01-16T12:00:00.000000000',\n '2015-02-15T00:00:00.000000000', '2015-03-16T12:00:00.000000000',\n '2015-04-16T00:00:00.000000000', '2015-05-16T12:00:00.000000000',\n '2015-06-16T00:00:00.000000000', '2015-07-16T12:00:00.000000000',\n '2015-08-16T12:00:00.000000000', '2015-09-16T00:00:00.000000000',\n '2015-10-16T12:00:00.000000000', '2015-11-16T00:00:00.000000000',\n '2015-12-16T12:00:00.000000000'], dtype='datetime64[ns]')Attributes: (7)comment :Dynamic sea surface height anomaly above the geoid, suitable for comparisons with altimetry sea surface height data products that apply the inverse barometer (IB) correction. Note: SSH is calculated by correcting model sea level anomaly ETAN for three effects: a) global mean steric sea level changes related to density changes in the Boussinesq volume-conserving model (Greatbatch correction, see sterGloH), b) the inverted barometer (IB) effect (see SSHIBC) and c) sea level displacement due to sea-ice and snow pressure loading (see sIceLoad). SSH can be compared with the similarly-named SSH variable in previous ECCO products that did not include atmospheric pressure loading (e.g., Version 4 Release 3). Use SSHNOIBC for comparisons with altimetry data products that do NOT apply the IB correction.coverage_content_type :modelResultlong_name :Dynamic sea surface height anomalystandard_name :sea_surface_height_above_geoidunits :mvalid_max :1.4207719564437866valid_min :-1.8805772066116333\n\n\n\n# Commenting for quarto site render\n# ds_multi['SSH'].hvplot.image()" + }, + { + "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#references", + "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#references", + "title": "PO.DAAC ECCO SSH", + "section": "References", + "text": "References\n\nhttps://github.com/fsspec/kerchunk\nhttps://medium.com/pangeo/fake-it-until-you-make-it-reading-goes-netcdf4-data-on-aws-s3-as-zarr-for-rapid-data-access-61e33f8fe685\nhttps://medium.com/pangeo/cloud-performant-reading-of-netcdf4-hdf5-data-using-the-zarr-library-1a95c5c92314" }, { "objectID": "workshops/setup.html", @@ -84,7 +315,105 @@ "text": "How do I end my session?\n(Also see How do I end my Openscapes session? Will I lose all of my work?)\nWhen you are finished working for the day it is important to explicitly log out of your Openscapes session. The reason for this is it will save us a bit of money! When you keep a session active it uses up AWS resources and keeps a series of virtual machines deployed.\nStopping the server happens automatically when you log out, so navigate to “File -> Log Out” and just click “Log Out”!\n!!! NOTE “logging out” - Logging out will NOT cause any of your work to be lost or deleted. It simply shuts down some resources. It would be equivalent to turning off your desktop computer at the end of the day." }, { - "objectID": "external/data_discovery_cmr.html", + "objectID": "external/data_discovery_cmr-stac_api.html", + "href": "external/data_discovery_cmr-stac_api.html", + "title": "Data Discovery with CMR-STAC API", + "section": "", + "text": "imported on: 2023-05-30\nThis notebook was originally developed for the 2021 Cloud Hackathon, and has been updated with most current approaches." + }, + { + "objectID": "external/data_discovery_cmr-stac_api.html#timing", + "href": "external/data_discovery_cmr-stac_api.html#timing", + "title": "Data Discovery with CMR-STAC API", + "section": "Timing", + "text": "Timing\n\nExercise: 30 min" + }, + { + "objectID": "external/data_discovery_cmr-stac_api.html#summary", + "href": "external/data_discovery_cmr-stac_api.html#summary", + "title": "Data Discovery with CMR-STAC API", + "section": "Summary", + "text": "Summary\nIn this example we will access the NASA’s Harmonized Landsat Sentinel-2 (HLS) version 2 assets, which are archived in cloud optimized geoTIFF (COG) format in the LP DAAC Cumulus cloud space. The COGs can be used like any other geoTIFF file, but have some added features that make them more efficient within the cloud data access paradigm. These features include: overviews and internal tiling. Below we will demonstrate how to leverage these features.\n\nBut first, what is STAC?\nSpatioTemporal Asset Catalog (STAC) is a specification that provides a common language for interpreting geospatial information in order to standardize indexing and discovering data.\nThe STAC specification is made up of a collection of related, yet independent specifications that when used together provide search and discovery capabilities for remote assets.\n\nFour STAC Specifications\nSTAC Catalog (aka DAAC Archive)\nSTAC Collection (aka Data Product)\nSTAC Item (aka Granule)\nSTAC API\nIn the following sections, we will explore each of STAC element using NASA’s Common Metadata Repository (CMR) STAC application programming interface (API), or CMR-STAC API for short.\n\n\n\nCMR-STAC API\nThe CMR-STAC API is NASA’s implementation of the STAC API specification for all NASA data holdings within EOSDIS. The current implementation does not allow for querries accross the entire NASA catalog. Users must execute searches within provider catalogs (e.g., LPCLOUD) to find the STAC Items they are searching for. All the providers can be found at the CMR-STAC endpoint here: https://cmr.earthdata.nasa.gov/stac/.\nIn this exercise, we will query the LPCLOUD provider to identify STAC Items from the Harmonized Landsat Sentinel-2 (HLS) collection that fall within our region of interest (ROI) and within our specified time range." + }, + { + "objectID": "external/data_discovery_cmr-stac_api.html#what-you-will-learn-from-this-tutorial", + "href": "external/data_discovery_cmr-stac_api.html#what-you-will-learn-from-this-tutorial", + "title": "Data Discovery with CMR-STAC API", + "section": "What you will learn from this tutorial", + "text": "What you will learn from this tutorial\n\nhow to connect to NASA CMR-STAC API using Python’s pystac-client\n\nhow to navigate CMR-STAC records\n\nhow to read in a geojson file using geopandas to specify your region of interest\nhow to use the CMR-STAC API to search for data\nhow to perform post-search filtering of CMR-STAC API search result in Python\n\nhow to extract and save data access URLs for geospatial assets\n\nThis exercise can be found in the 2021 Cloud Hackathon Book" + }, + { + "objectID": "external/data_discovery_cmr-stac_api.html#import-required-packages", + "href": "external/data_discovery_cmr-stac_api.html#import-required-packages", + "title": "Data Discovery with CMR-STAC API", + "section": "Import Required Packages", + "text": "Import Required Packages\n\nfrom pystac_client import Client \nfrom collections import defaultdict \nimport json\nimport geopandas\nimport geoviews as gv\nfrom cartopy import crs\ngv.extension('bokeh', 'matplotlib')" + }, + { + "objectID": "external/data_discovery_cmr-stac_api.html#explored-available-nasa-providers", + "href": "external/data_discovery_cmr-stac_api.html#explored-available-nasa-providers", + "title": "Data Discovery with CMR-STAC API", + "section": "Explored available NASA Providers", + "text": "Explored available NASA Providers\n\nSTAC_URL = 'https://cmr.earthdata.nasa.gov/stac'\n\n\nConnect to the CMR-STAC API\n\nprovider_cat = Client.open(STAC_URL)\n\nWe’ll create a providers variable so we can take a deeper look into available data providers - subcategories are referred to as “children”. We can then print them as a for loop.\n\nproviders = [p for p in provider_cat.get_children()]\n\nfor count, provider in enumerate(providers):\n print(f'{count} - {provider.title}')\n\n0 - LARC_ASDC\n1 - USGS_EROS\n2 - ESA\n3 - GHRC\n4 - LAADS\n5 - OBPG\n6 - OB_DAAC\n7 - ECHO\n8 - ISRO\n9 - LPCUMULUS\n10 - EDF_DEV04\n11 - GES_DISC\n12 - ASF\n13 - OMINRT\n14 - EUMETSAT\n15 - NCCS\n16 - NSIDCV0\n17 - PODAAC\n18 - LARC\n19 - USGS\n20 - SCIOPS\n21 - LANCEMODIS\n22 - CDDIS\n23 - JAXA\n24 - AU_AADC\n25 - ECHO10_OPS\n26 - LPDAAC_ECS\n27 - NSIDC_ECS\n28 - ORNL_DAAC\n29 - LM_FIRMS\n30 - SEDAC\n31 - LANCEAMSR2\n32 - NOAA_NCEI\n33 - USGS_LTA\n34 - GESDISCCLD\n35 - GHRSSTCWIC\n36 - ASIPS\n37 - ESDIS\n38 - POCLOUD\n39 - NSIDC_CPRD\n40 - ORNL_CLOUD\n41 - FEDEO\n42 - XYZ_PROV\n43 - GHRC_DAAC\n44 - CSDA\n45 - NRSCC\n46 - CEOS_EXTRA\n47 - MOPITT\n48 - GHRC_CLOUD\n49 - LPCLOUD\n50 - CCMEO" + }, + { + "objectID": "external/data_discovery_cmr-stac_api.html#connect-to-the-lpcloud-providerstac-catalog", + "href": "external/data_discovery_cmr-stac_api.html#connect-to-the-lpcloud-providerstac-catalog", + "title": "Data Discovery with CMR-STAC API", + "section": "Connect to the LPCLOUD Provider/STAC Catalog", + "text": "Connect to the LPCLOUD Provider/STAC Catalog\nFor this next step we need the provider title (e.g., LPCLOUD) from above. We will add the provider to the end of the CMR-STAC API URL (i.e., https://cmr.earthdata.nasa.gov/stac/) to connect to the LPCLOUD STAC Catalog.\n\ncatalog = Client.open(f'{STAC_URL}/LPCLOUD/')\n\nSince we are using a dedicated client (i.e., pystac-client.Client) to connect to our STAC Provider Catalog, we will have access to some useful internal methods and functions (e.g., get_children() or get_all_items()) we can use to get information from these objects." + }, + { + "objectID": "external/data_discovery_cmr-stac_api.html#list-stac-collections", + "href": "external/data_discovery_cmr-stac_api.html#list-stac-collections", + "title": "Data Discovery with CMR-STAC API", + "section": "List STAC Collections", + "text": "List STAC Collections\nWe’ll create a products variable to view deeper in the STAC Catalog.\n\nproducts = [c for c in catalog.get_children()]\n\n\nPrint one of the STAC Collection records\nTo view the products variable we just created, let’s look at one entry as a dictionary.\n\nproducts[1].to_dict()\n\n{'type': 'Collection',\n 'id': 'HLSL30.v2.0',\n 'stac_version': '1.0.0',\n 'description': 'The Harmonized Landsat and Sentinel-2 (HLS) project provides consistent surface reflectance (SR) and top of atmosphere (TOA) brightness data from the Operational Land Imager (OLI) aboard the joint NASA/USGS Landsat 8 satellite and the Multi-Spectral Instrument (MSI) aboard Europe’s Copernicus Sentinel-2A and Sentinel-2B satellites. The combined measurement enables global observations of the land every 2–3 days at 30-meter (m) spatial resolution. The HLS project uses a set of algorithms to obtain seamless products from OLI and MSI that include atmospheric correction, cloud and cloud-shadow masking, spatial co-registration and common gridding, illumination and view angle normalization, and spectral bandpass adjustment.\\r\\n\\r\\nThe HLSL30 product provides 30-m Nadir Bidirectional Reflectance Distribution Function (BRDF)-Adjusted Reflectance (NBAR) and is derived from Landsat 8 OLI data products. The HLSS30 and HLSL30 products are gridded to the same resolution and Military Grid Reference System ([MGRS](https://hls.gsfc.nasa.gov/products-description/tiling-system/)) tiling system, and thus are “stackable” for time series analysis.\\r\\n\\r\\nThe HLSL30 product is provided in Cloud Optimized GeoTIFF (COG) format, and each band is distributed as a separate file. There are 11 bands included in the HLSL30 product along with one quality assessment (QA) band and four angle bands. See the User Guide for a more detailed description of the individual bands provided in the HLSL30 product.',\n 'links': [{'rel': <RelType.ROOT: 'root'>,\n 'href': 'https://cmr.earthdata.nasa.gov/stac/',\n 'type': <MediaType.JSON: 'application/json'>},\n {'rel': 'items',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0/items',\n 'type': 'application/json',\n 'title': 'Granules in this collection'},\n {'rel': 'about',\n 'href': 'https://cmr.earthdata.nasa.gov/search/concepts/C2021957657-LPCLOUD.html',\n 'type': 'text/html',\n 'title': 'HTML metadata for collection'},\n {'rel': 'via',\n 'href': 'https://cmr.earthdata.nasa.gov/search/concepts/C2021957657-LPCLOUD.json',\n 'type': 'application/json',\n 'title': 'CMR JSON metadata for collection'},\n {'rel': 'child',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0/2020',\n 'type': 'application/json',\n 'title': '2020 catalog'},\n {'rel': 'child',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0/2021',\n 'type': 'application/json',\n 'title': '2021 catalog'},\n {'rel': <RelType.SELF: 'self'>,\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0',\n 'type': <MediaType.JSON: 'application/json'>},\n {'rel': <RelType.PARENT: 'parent'>,\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/',\n 'type': <MediaType.JSON: 'application/json'>}],\n 'stac_extensions': [],\n 'title': 'HLS Landsat Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0',\n 'extent': {'spatial': {'bbox': [[-180, -90, 180, 90]]},\n 'temporal': {'interval': [['2013-05-01T00:00:00Z', None]]}},\n 'license': 'not-provided'}\n\n\n\n\nPrint the STAC Collection ids with their title\nIn the above output, id and title are two elements of interest that we can print for all products using a for loop.\n\nfor p in products: \n print(f\"{p.id}: {p.title}\")\n\nASTGTM.v003: ASTER Global Digital Elevation Model V003\nHLSL30.v2.0: HLS Landsat Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0\nHLSL30.v1.5: HLS Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30 m V1.5\nHLSS30.v1.5: HLS Sentinel-2 Multi-spectral Instrument Surface Reflectance Daily Global 30 m V1.5\nHLSS30.v2.0: HLS Sentinel-2 Multi-spectral Instrument Surface Reflectance Daily Global 30m v2.0" + }, + { + "objectID": "external/data_discovery_cmr-stac_api.html#search-for-granulesstac-items---set-up-query-parameters-to-submit-to-the-cmr-stac-api", + "href": "external/data_discovery_cmr-stac_api.html#search-for-granulesstac-items---set-up-query-parameters-to-submit-to-the-cmr-stac-api", + "title": "Data Discovery with CMR-STAC API", + "section": "Search for Granules/STAC Items - Set up query parameters to submit to the CMR-STAC API", + "text": "Search for Granules/STAC Items - Set up query parameters to submit to the CMR-STAC API\nWe will define our ROI using a geojson file containing a small polygon feature in western Nebraska, USA. The geojson file is found in the ~/data directory. We’ll also specify the data collections and a time range for our example.\n\nRead in a geojson file\nReading in a geojson file with geopandas will return the geometry of our polygon (our ROI).\nNOTE: If you are running the notebook from the tutorials-templates directory, please use the following path to connect to the geojson file: “../tutorials/data/ne_w_agfields.geojson”\n\nfield = geopandas.read_file('./data/ne_w_agfields.geojson')\nfield\n\n\n\n\n\n\n\n\ngeometry\n\n\n\n\n0\nPOLYGON ((-101.67272 41.04754, -101.65345 41.0...\n\n\n\n\n\n\n\n\n\nVisualize contents of geojson file\nWe can use that geometry to visualize the polygon: here, a square. But wait for it –\n\nfieldShape = field['geometry'][0]\nfieldShape\n\n\n\n\nWe can plot the polygon using the geoviews package that we imported as gv with ‘bokeh’ and ‘matplotlib’ extensions. The following has reasonable width, height, color, and line widths to view our polygon when it is overlayed on a base tile map.\n\nbase = gv.tile_sources.EsriImagery.opts(width=650, height=500)\nfarmField = gv.Polygons(fieldShape).opts(line_color='yellow', line_width=10, color=None)\nbase * farmField\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\nWe will now start to specify the search criteria we are interested in, i.e, the date range, the ROI, and the data collections, that we will pass to the STAC API.\n\n\nExtract the coordinates for the ROI\n\nroi = json.loads(field.to_json())['features'][0]['geometry']\nroi\n\n{'type': 'Polygon',\n 'coordinates': [[[-101.67271614074707, 41.04754380304359],\n [-101.65344715118408, 41.04754380304359],\n [-101.65344715118408, 41.06213891056728],\n [-101.67271614074707, 41.06213891056728],\n [-101.67271614074707, 41.04754380304359]]]}\n\n\nSo, what just happen there? Let’s take a quick detour to break it down.\n\n\n\nSpecify date range\nNext up is to specify our date range using ISO_8601 date formatting.\n\n#date_range = \"2021-05-01T00:00:00Z/2021-08-30T23:59:59Z\" # closed interval\n#date_range = \"2021-05-01T00:00:00Z/..\" # open interval - does not currently work with the CMR-STAC API\ndate_range = \"2021-05/2021-08\"\n\n\n\nSpecify the STAC Collections\nSTAC Collection is synonomous with what we usually consider a NASA data product. Desired STAC Collections are submitted to the search API as a list containing the collection id. We can use the ids that we printed from our products for loop above. Let’s focus on S30 and L30 collections.\n\ncollections = ['HLSL30.v2.0', 'HLSS30.v2.0']\ncollections\n\n['HLSL30.v2.0', 'HLSS30.v2.0']" + }, + { + "objectID": "external/data_discovery_cmr-stac_api.html#search-the-cmr-stac-api-with-our-search-criteria", + "href": "external/data_discovery_cmr-stac_api.html#search-the-cmr-stac-api-with-our-search-criteria", + "title": "Data Discovery with CMR-STAC API", + "section": "Search the CMR-STAC API with our search criteria", + "text": "Search the CMR-STAC API with our search criteria\nNow we can put all our search criteria together using catalog.search from the pystac_client package.\n\nsearch = catalog.search(\n collections=collections,\n intersects=roi,\n datetime=date_range,\n limit=100\n)\n\n\nPrint out how many STAC Items match our search query\n\nsearch.matched()\n\n113\n\n\nWe now have a search object containing the STAC Items that matched our query. Now, let’s pull out all of the STAC Items (as a PySTAC ItemCollection object) and explore the contents (i.e., the STAC Items)\n\nitem_collection = search.get_all_items()\n\nLet’s list some of the Items from our pystac item_collection:\n\nlist(item_collection)[0:5]\n\n[<Item id=HLS.L30.T13TGF.2021124T173013.v2.0>,\n <Item id=HLS.L30.T14TKL.2021124T173013.v2.0>,\n <Item id=HLS.S30.T14TKL.2021125T172901.v2.0>,\n <Item id=HLS.S30.T13TGF.2021125T172901.v2.0>,\n <Item id=HLS.S30.T14TKL.2021128T173901.v2.0>]\n\n\nWe can view a single Item as a dictionary, as we did above with STAC Collections/products.\n\nitem_collection[0].to_dict()\n\n{'type': 'Feature',\n 'stac_version': '1.0.0',\n 'id': 'HLS.L30.T13TGF.2021124T173013.v2.0',\n 'properties': {'datetime': '2021-05-04T17:30:13.428000Z',\n 'start_datetime': '2021-05-04T17:30:13.428Z',\n 'end_datetime': '2021-05-04T17:30:37.319Z',\n 'eo:cloud_cover': 36},\n 'geometry': {'type': 'Polygon',\n 'coordinates': [[[-101.5423534, 40.5109845],\n [-101.3056118, 41.2066375],\n [-101.2894253, 41.4919436],\n [-102.6032964, 41.5268623],\n [-102.638891, 40.5386175],\n [-101.5423534, 40.5109845]]]},\n 'links': [{'rel': 'self',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0/items/HLS.L30.T13TGF.2021124T173013.v2.0'},\n {'rel': 'parent',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0'},\n {'rel': 'collection',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0'},\n {'rel': <RelType.ROOT: 'root'>,\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/',\n 'type': <MediaType.JSON: 'application/json'>},\n {'rel': 'provider', 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD'},\n {'rel': 'via',\n 'href': 'https://cmr.earthdata.nasa.gov/search/concepts/G2144020713-LPCLOUD.json'},\n {'rel': 'via',\n 'href': 'https://cmr.earthdata.nasa.gov/search/concepts/G2144020713-LPCLOUD.umm_json'}],\n 'assets': {'B11': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B11.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B11.tif'},\n 'B07': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B07.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B07.tif'},\n 'SAA': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.SAA.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.SAA.tif'},\n 'B06': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B06.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B06.tif'},\n 'B09': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B09.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B09.tif'},\n 'B10': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B10.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B10.tif'},\n 'VZA': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.VZA.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.VZA.tif'},\n 'SZA': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.SZA.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.SZA.tif'},\n 'B01': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B01.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B01.tif'},\n 'VAA': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.VAA.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.VAA.tif'},\n 'B05': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B05.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B05.tif'},\n 'B02': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B02.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B02.tif'},\n 'Fmask': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.Fmask.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.Fmask.tif'},\n 'B03': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B03.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B03.tif'},\n 'B04': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B04.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B04.tif'},\n 'browse': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.jpg',\n 'type': 'image/jpeg',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.jpg'},\n 'metadata': {'href': 'https://cmr.earthdata.nasa.gov/search/concepts/G2144020713-LPCLOUD.xml',\n 'type': 'application/xml'}},\n 'bbox': [-102.638891, 40.510984, -101.289425, 41.526862],\n 'stac_extensions': ['https://stac-extensions.github.io/eo/v1.0.0/schema.json'],\n 'collection': 'HLSL30.v2.0'}" + }, + { + "objectID": "external/data_discovery_cmr-stac_api.html#filtering-stac-items", + "href": "external/data_discovery_cmr-stac_api.html#filtering-stac-items", + "title": "Data Discovery with CMR-STAC API", + "section": "Filtering STAC Items", + "text": "Filtering STAC Items\nWhile the CMR-STAC API is a powerful search and discovery utility, it is still maturing and currently does not have the full gamut of filtering capabilities that the STAC API specification allows for. Hence, additional filtering is required if we want to filter by a property, for example cloud cover. Below we will loop through and filter the item_collection by a specified cloud cover as well as extract the band we’d need to do an Enhanced Vegetation Index (EVI) calculation for a future analysis.\nWe’ll make a cloudcover variable where we will set the maximum allowable cloud cover and extract the band links for those Items that match or are less than the max cloud cover.\n\ncloudcover = 25\n\nWe will also specify the STAC Assets (i.e., bands/layers) of interest for both the S30 and L30 collections (also in our collections variable above).\nIn this hypothetical workflow, we’ll extract the bands needed to calculate an enhanced vegetation index (EVI). Thus, the band needed include red, near infrared (NIR), and blue. We’ll also extract a quality band (i.e., Fmask) that we’d eventually use to perform per-pixel quality filtering.\nNotice that the band ids are in some case not one-to-one between the S30 and the L30 product. This is evident in the NIR band for each product where S30’s NIR band id is B8A and L30’s is B05. Note, the S30 product has an additional NIR band with a band id of B08, but the spectral ranges between B8A and B05 are more closely aligned. Visit the HLS Overview page to learn more about HLS spectral bands.\n\ns30_bands = ['B8A', 'B04', 'B02', 'Fmask'] # S30 bands for EVI calculation and quality filtering -> NIR, RED, BLUE, Quality \nl30_bands = ['B05', 'B04', 'B02', 'Fmask'] # L30 bands for EVI calculation and quality filtering -> NIR, RED, BLUE, Quality \n\nAnd now to loop through and filter the item_collection by cloud cover and bands:\n\nevi_band_links = []\n\nfor i in item_collection:\n if i.properties['eo:cloud_cover'] <= cloudcover:\n if i.collection_id == 'HLSS30.v2.0':\n #print(i.properties['eo:cloud_cover'])\n evi_bands = s30_bands\n elif i.collection_id == 'HLSL30.v2.0':\n #print(i.properties['eo:cloud_cover'])\n evi_bands = l30_bands\n\n for a in i.assets:\n if any(b==a for b in evi_bands):\n evi_band_links.append(i.assets[a].href)\n\nThe filtering done in the previous steps produces a list of links to STAC Assets. Let’s print out the first ten links.\n\nevi_band_links[:10]\n\n['https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B05.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.Fmask.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B02.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T14TKL.2021133T172406.v2.0/HLS.L30.T14TKL.2021133T172406.v2.0.B02.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T14TKL.2021133T172406.v2.0/HLS.L30.T14TKL.2021133T172406.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T14TKL.2021133T172406.v2.0/HLS.L30.T14TKL.2021133T172406.v2.0.B05.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T14TKL.2021133T172406.v2.0/HLS.L30.T14TKL.2021133T172406.v2.0.Fmask.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T14TKL.2021133T173859.v2.0/HLS.S30.T14TKL.2021133T173859.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T14TKL.2021133T173859.v2.0/HLS.S30.T14TKL.2021133T173859.v2.0.B8A.tif']\n\n\nNOTE that HLS data is mapped to the Universal Transverse Mercator (UTM) projection and is tiled using the Sentinel-2 Military Grid Reference System (MGRS) UTM grid. Notice that in the list of links we have multiple tiles, i.e. T14TKL & T13TGF, that intersect with our region of interest. In this case, these two tiles represent neighboring UTM zones. The tiles can be discern from the file name, which is the last element in a link (far right) following the last forward slash (/) - e.g., HLS.L30.T14TKL.2021133T172406.v1.5.B04.tif. The figure below explains where to find the tile/UTM zone from the file name.\n\nWe will now split the list of links into separate logical sub-lists." + }, + { + "objectID": "external/data_discovery_cmr-stac_api.html#split-data-links-list-into-logical-groupings", + "href": "external/data_discovery_cmr-stac_api.html#split-data-links-list-into-logical-groupings", + "title": "Data Discovery with CMR-STAC API", + "section": "Split Data Links List into Logical Groupings", + "text": "Split Data Links List into Logical Groupings\nWe have a list of links to data assets that meet our search and filtering criteria. Below we’ll split our list from above into lists first by tile/UTM zone and then further by individual bands bands. The commands that follow will do the splitting with python routines.\n\nSplit by UTM tile specified in the file name (e.g., T14TKL & T13TGF)\n\ntile_dicts = defaultdict(list) # https://stackoverflow.com/questions/26367812/appending-to-list-in-python-dictionary\n\n\nfor l in evi_band_links:\n tile = l.split('.')[-6]\n tile_dicts[tile].append(l)\n\n\nPrint dictionary keys and values, i.e. the data links\n\ntile_dicts.keys()\n\ndict_keys(['T13TGF', 'T14TKL'])\n\n\n\ntile_dicts['T13TGF'][:5]\n\n['https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B05.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.Fmask.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B02.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021133T173859.v2.0/HLS.S30.T13TGF.2021133T173859.v2.0.B8A.tif']\n\n\nNow we will create a separate list of data links for each tile\n\ntile_links_T14TKL = tile_dicts['T14TKL']\ntile_links_T13TGF = tile_dicts['T13TGF']\n\n\n\nPrint band/layer links for HLS tile T13TGF\n\n# tile_links_T13TGF[:10]\n\n\n\n\nSplit the links by band\n\nbands_dicts = defaultdict(list)\n\n\nfor b in tile_links_T13TGF:\n band = b.split('.')[-2]\n bands_dicts[band].append(b)\n\n\nbands_dicts.keys()\n\ndict_keys(['B04', 'B05', 'Fmask', 'B02', 'B8A'])\n\n\n\nbands_dicts['B04']\n\n['https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021133T173859.v2.0/HLS.S30.T13TGF.2021133T173859.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021140T173021.v2.0/HLS.L30.T13TGF.2021140T173021.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021140T172859.v2.0/HLS.S30.T13TGF.2021140T172859.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021145T172901.v2.0/HLS.S30.T13TGF.2021145T172901.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021155T172901.v2.0/HLS.S30.T13TGF.2021155T172901.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021156T173029.v2.0/HLS.L30.T13TGF.2021156T173029.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021158T173901.v2.0/HLS.S30.T13TGF.2021158T173901.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021163T173909.v2.0/HLS.S30.T13TGF.2021163T173909.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021165T172422.v2.0/HLS.L30.T13TGF.2021165T172422.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021165T172901.v2.0/HLS.S30.T13TGF.2021165T172901.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021173T173909.v2.0/HLS.S30.T13TGF.2021173T173909.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021185T172901.v2.0/HLS.S30.T13TGF.2021185T172901.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021188T173037.v2.0/HLS.L30.T13TGF.2021188T173037.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021190T172859.v2.0/HLS.S30.T13TGF.2021190T172859.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021193T173909.v2.0/HLS.S30.T13TGF.2021193T173909.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021198T173911.v2.0/HLS.S30.T13TGF.2021198T173911.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021200T172859.v2.0/HLS.S30.T13TGF.2021200T172859.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021203T173909.v2.0/HLS.S30.T13TGF.2021203T173909.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021204T173042.v2.0/HLS.L30.T13TGF.2021204T173042.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021208T173911.v2.0/HLS.S30.T13TGF.2021208T173911.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021210T172859.v2.0/HLS.S30.T13TGF.2021210T172859.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021215T172901.v2.0/HLS.S30.T13TGF.2021215T172901.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021218T173911.v2.0/HLS.S30.T13TGF.2021218T173911.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021220T173049.v2.0/HLS.L30.T13TGF.2021220T173049.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021220T172859.v2.0/HLS.S30.T13TGF.2021220T172859.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021223T173909.v2.0/HLS.S30.T13TGF.2021223T173909.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021228T173911.v2.0/HLS.S30.T13TGF.2021228T173911.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021229T172441.v2.0/HLS.L30.T13TGF.2021229T172441.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021230T172859.v2.0/HLS.S30.T13TGF.2021230T172859.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021235T172901.v2.0/HLS.S30.T13TGF.2021235T172901.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021243T173859.v2.0/HLS.S30.T13TGF.2021243T173859.v2.0.B04.tif']" + }, + { + "objectID": "external/data_discovery_cmr-stac_api.html#save-links-to-a-text-file", + "href": "external/data_discovery_cmr-stac_api.html#save-links-to-a-text-file", + "title": "Data Discovery with CMR-STAC API", + "section": "Save links to a text file", + "text": "Save links to a text file\nTo complete this exercise, we will save the individual link lists as separate text files with descriptive names.\nNOTE: If you are running the notebook from the tutorials-templates directory, please use the following path to write to the data directory: “../tutorials/data/{name}”\n\nWrite links from CMR-STAC API to a file\n\nfor k, v in bands_dicts.items():\n name = (f'HTTPS_T13TGF_{k}_Links.txt')\n with open(f'./data/{name}', 'w') as f: # use ../tutorials/data/{name} as your path if running the notebook from \"tutorials-template\"\n for l in v:\n f.write(f\"{l}\" + '\\n')\n\n\n\nWrite links to file for S3 access\n\nfor k, v in bands_dicts.items():\n name = (f'S3_T13TGF_{k}_Links.txt')\n with open(f'./data/{name}', 'w') as f: # use ../tutorials/data/{name} as your path if running the notebook from \"tutorials-template\"\n for l in v:\n s3l = l.replace('https://data.lpdaac.earthdatacloud.nasa.gov/', 's3://')\n f.write(f\"{s3l}\" + '\\n')" + }, + { + "objectID": "external/data_discovery_cmr-stac_api.html#resources", + "href": "external/data_discovery_cmr-stac_api.html#resources", + "title": "Data Discovery with CMR-STAC API", + "section": "Resources", + "text": "Resources\n\nSTAC Specification Webpage\nSTAC API Documentation\nCMR-STAC API Github\nPySTAC Client Documentation\nhttps://stackoverflow.com/questions/26367812/appending-to-list-in-python-dictionary\nGeopandas\nHLS Overview" + }, + { + "objectID": "external/data_discovery_cmr.html", "href": "external/data_discovery_cmr.html", "title": "Data discovery with CMR", "section": "", @@ -119,67 +448,74 @@ "text": "Granule Search\nIn NASA speak, Granules are files. In this example, we will search for recent Sentinel-1 Ground Range Detected (GRD) Medium Resolution Synthetic Aperture Radar images over the east coast of Greenland. The data in these files are most useful for sea ice mapping.\nI’ll use the data range 2021-10-17 00:00 to 2021-10-18 23:59:59.\nI’ll use a simple bounding box to search. - SW: 76.08166,-67.1746 - NW: 88.19689,21.04862\nFrom the collections search, I know the concept ids for Sentinel-1A and Sentinel-1B GRD medium resolution are - C1214472336-ASF - C1327985578-ASF\nWe need to change the resource url to look for granules instead of collections\n\nurl = f'{CMR_OPS}/{\"granules\"}'\n\nWe will search by concept_id, temporal, and bounding_box. Details about these search parameters can be found in the CMR API Documentation.\nThe formatting of the values for each parameter is quite specific.\nTemporal parameters are in ISO 8061 format yyyy-MM-ddTHH:mm:ssZ.\nBounding box coordinates are lower left longitude, lower left latitude, upper right longitude, upper right latitude.\n\nresponse = requests.get(url, \n params={\n 'concept_id': 'C1214472336-ASF',\n 'temporal': '2020-10-17T00:00:00Z,2020-10-18T23:59:59Z',\n 'bounding_box': '76.08166,-67.1746,88.19689,21.04862',\n 'page_size': 200,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\nprint(response.status_code)\n\n200\n\n\n\nprint(response.headers['CMR-Hits'])\n\n6\n\n\n\ngranules = response.json()['feed']['entry']\n#for granule in granules:\n# print(f'{granule[\"archive_center\"]} {granule[\"dataset_id\"]} {granule[\"id\"]}')\n\n\npprint(granules)\n\n[{'browse_flag': True,\n 'collection_concept_id': 'C1214472336-ASF',\n 'coordinate_system': 'GEODETIC',\n 'data_center': 'ASF',\n 'dataset_id': 'SENTINEL-1A_DUAL_POL_METADATA_GRD_MEDIUM_RES',\n 'day_night_flag': 'UNSPECIFIED',\n 'granule_size': '0.05633258819580078',\n 'id': 'G1954601581-ASF',\n 'links': [{'href': 'https://datapool.asf.alaska.edu/METADATA_GRD_MD/SA/S1A_EW_GRDM_1SDH_20201017T132009_20201017T132039_034836_040F98_404E.iso.xml',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',\n 'title': 'This link provides direct download access to the '\n 'granule.'},\n {'href': 'www.asf.alaska.edu/sar-data-sets/sentinel-1',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#',\n 'title': 'ASF DAAC Sentinel-1 data set landing page (VIEW RELATED '\n 'INFORMATION)'},\n {'href': 'www.asf.alaska.edu/sar-information/sentinel-1-documents-tools',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#',\n 'title': 'ASF DAAC Sentinel-1 User Guide and Technical '\n 'Documentation (VIEW RELATED INFORMATION)'},\n {'href': 'https://datapool.asf.alaska.edu/BROWSE/SA/S1A_EW_GRDM_1SDH_20201017T132009_20201017T132039_034836_040F98_404E.jpg',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/browse#'},\n {'href': 'https://vertex.daac.asf.alaska.edu/',\n 'hreflang': 'en-US',\n 'inherited': True,\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#'}],\n 'online_access_flag': True,\n 'orbit_calculated_spatial_domains': [{'orbit_number': '34836'}],\n 'original_format': 'ECHO10',\n 'polygons': [['-59.163563 87.942726 -60.893669 89.293564 -59.279579 '\n '96.119583 -57.619923 94.49958 -59.163563 87.942726']],\n 'producer_granule_id': 'S1A_EW_GRDM_1SDH_20201017T132009_20201017T132039_034836_040F98_404E',\n 'time_end': '2020-10-17T13:20:39.000Z',\n 'time_start': '2020-10-17T13:20:09.000Z',\n 'title': 'S1A_EW_GRDM_1SDH_20201017T132009_20201017T132039_034836_040F98_404E-METADATA_GRD_MD',\n 'updated': '2020-10-19T17:13:39.000Z'},\n {'browse_flag': False,\n 'collection_concept_id': 'C1214472336-ASF',\n 'coordinate_system': 'GEODETIC',\n 'data_center': 'ASF',\n 'dataset_id': 'SENTINEL-1A_DUAL_POL_METADATA_GRD_MEDIUM_RES',\n 'day_night_flag': 'UNSPECIFIED',\n 'granule_size': '0.05633354187011719',\n 'id': 'G1954616816-ASF',\n 'links': [{'href': 'https://datapool.asf.alaska.edu/METADATA_GRD_MD/SA/S1A_EW_GRDM_1SDH_20201017T145616_20201017T145720_034837_040FA0_0B4B.iso.xml',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',\n 'title': 'This link provides direct download access to the '\n 'granule.'},\n {'href': 'www.asf.alaska.edu/sar-data-sets/sentinel-1',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#',\n 'title': 'ASF DAAC Sentinel-1 data set landing page (VIEW RELATED '\n 'INFORMATION)'},\n {'href': 'www.asf.alaska.edu/sar-information/sentinel-1-documents-tools',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#',\n 'title': 'ASF DAAC Sentinel-1 User Guide and Technical '\n 'Documentation (VIEW RELATED INFORMATION)'},\n {'href': 'https://vertex.daac.asf.alaska.edu/',\n 'hreflang': 'en-US',\n 'inherited': True,\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#'}],\n 'online_access_flag': True,\n 'orbit_calculated_spatial_domains': [{'orbit_number': '34837'}],\n 'original_format': 'ECHO10',\n 'polygons': [['-66.104271 69.819366 -69.571243 74.741966 -67.42112 83.209152 '\n '-64.210938 77.59269 -66.104271 69.819366']],\n 'producer_granule_id': 'S1A_EW_GRDM_1SDH_20201017T145616_20201017T145720_034837_040FA0_0B4B',\n 'time_end': '2020-10-17T14:57:20.000Z',\n 'time_start': '2020-10-17T14:56:16.000Z',\n 'title': 'S1A_EW_GRDM_1SDH_20201017T145616_20201017T145720_034837_040FA0_0B4B-METADATA_GRD_MD',\n 'updated': '2020-10-19T18:35:34.000Z'},\n {'browse_flag': True,\n 'collection_concept_id': 'C1214472336-ASF',\n 'coordinate_system': 'GEODETIC',\n 'data_center': 'ASF',\n 'dataset_id': 'SENTINEL-1A_DUAL_POL_METADATA_GRD_MEDIUM_RES',\n 'day_night_flag': 'UNSPECIFIED',\n 'granule_size': '0.05633354187011719',\n 'id': 'G1954616638-ASF',\n 'links': [{'href': 'https://datapool.asf.alaska.edu/METADATA_GRD_MD/SA/S1A_EW_GRDM_1SDH_20201017T145720_20201017T145820_034837_040FA0_72CE.iso.xml',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',\n 'title': 'This link provides direct download access to the '\n 'granule.'},\n {'href': 'www.asf.alaska.edu/sar-data-sets/sentinel-1',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#',\n 'title': 'ASF DAAC Sentinel-1 data set landing page (VIEW RELATED '\n 'INFORMATION)'},\n {'href': 'www.asf.alaska.edu/sar-information/sentinel-1-documents-tools',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#',\n 'title': 'ASF DAAC Sentinel-1 User Guide and Technical '\n 'Documentation (VIEW RELATED INFORMATION)'},\n {'href': 'https://datapool.asf.alaska.edu/BROWSE/SA/S1A_EW_GRDM_1SDH_20201017T145720_20201017T145820_034837_040FA0_72CE.jpg',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/browse#'},\n {'href': 'https://vertex.daac.asf.alaska.edu/',\n 'hreflang': 'en-US',\n 'inherited': True,\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#'}],\n 'online_access_flag': True,\n 'orbit_calculated_spatial_domains': [{'orbit_number': '34837'}],\n 'original_format': 'ECHO10',\n 'polygons': [['-62.765087 66.277176 -66.103951 69.81897 -64.211227 77.590164 '\n '-61.060097 73.427185 -62.765087 66.277176']],\n 'producer_granule_id': 'S1A_EW_GRDM_1SDH_20201017T145720_20201017T145820_034837_040FA0_72CE',\n 'time_end': '2020-10-17T14:58:20.000Z',\n 'time_start': '2020-10-17T14:57:20.000Z',\n 'title': 'S1A_EW_GRDM_1SDH_20201017T145720_20201017T145820_034837_040FA0_72CE-METADATA_GRD_MD',\n 'updated': '2020-10-19T18:33:31.000Z'},\n {'browse_flag': True,\n 'collection_concept_id': 'C1214472336-ASF',\n 'coordinate_system': 'GEODETIC',\n 'data_center': 'ASF',\n 'dataset_id': 'SENTINEL-1A_DUAL_POL_METADATA_GRD_MEDIUM_RES',\n 'day_night_flag': 'UNSPECIFIED',\n 'granule_size': '0.05633354187011719',\n 'id': 'G1954805829-ASF',\n 'links': [{'href': 'https://datapool.asf.alaska.edu/METADATA_GRD_MD/SA/S1A_EW_GRDM_1SDH_20201018T135856_20201018T140000_034851_041027_2659.iso.xml',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',\n 'title': 'This link provides direct download access to the '\n 'granule.'},\n {'href': 'www.asf.alaska.edu/sar-data-sets/sentinel-1',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#',\n 'title': 'ASF DAAC Sentinel-1 data set landing page (VIEW RELATED '\n 'INFORMATION)'},\n {'href': 'www.asf.alaska.edu/sar-information/sentinel-1-documents-tools',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#',\n 'title': 'ASF DAAC Sentinel-1 User Guide and Technical '\n 'Documentation (VIEW RELATED INFORMATION)'},\n {'href': 'https://datapool.asf.alaska.edu/BROWSE/SA/S1A_EW_GRDM_1SDH_20201018T135856_20201018T140000_034851_041027_2659.jpg',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/browse#'},\n {'href': 'https://vertex.daac.asf.alaska.edu/',\n 'hreflang': 'en-US',\n 'inherited': True,\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#'}],\n 'online_access_flag': True,\n 'orbit_calculated_spatial_domains': [{'orbit_number': '34851'}],\n 'original_format': 'ECHO10',\n 'polygons': [['-65.236397 83.18071 -68.712318 87.887711 -66.630493 96.161102 '\n '-63.400326 90.779785 -65.236397 83.18071']],\n 'producer_granule_id': 'S1A_EW_GRDM_1SDH_20201018T135856_20201018T140000_034851_041027_2659',\n 'time_end': '2020-10-18T14:00:00.000Z',\n 'time_start': '2020-10-18T13:58:56.000Z',\n 'title': 'S1A_EW_GRDM_1SDH_20201018T135856_20201018T140000_034851_041027_2659-METADATA_GRD_MD',\n 'updated': '2020-10-20T07:16:54.000Z'},\n {'browse_flag': True,\n 'collection_concept_id': 'C1214472336-ASF',\n 'coordinate_system': 'GEODETIC',\n 'data_center': 'ASF',\n 'dataset_id': 'SENTINEL-1A_DUAL_POL_METADATA_GRD_MEDIUM_RES',\n 'day_night_flag': 'UNSPECIFIED',\n 'granule_size': '0.05633354187011719',\n 'id': 'G1954799806-ASF',\n 'links': [{'href': 'https://datapool.asf.alaska.edu/METADATA_GRD_MD/SA/S1A_EW_GRDM_1SDH_20201018T140000_20201018T140100_034851_041027_2777.iso.xml',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',\n 'title': 'This link provides direct download access to the '\n 'granule.'},\n {'href': 'www.asf.alaska.edu/sar-data-sets/sentinel-1',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#',\n 'title': 'ASF DAAC Sentinel-1 data set landing page (VIEW RELATED '\n 'INFORMATION)'},\n {'href': 'www.asf.alaska.edu/sar-information/sentinel-1-documents-tools',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#',\n 'title': 'ASF DAAC Sentinel-1 User Guide and Technical '\n 'Documentation (VIEW RELATED INFORMATION)'},\n {'href': 'https://datapool.asf.alaska.edu/BROWSE/SA/S1A_EW_GRDM_1SDH_20201018T140000_20201018T140100_034851_041027_2777.jpg',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/browse#'},\n {'href': 'https://vertex.daac.asf.alaska.edu/',\n 'hreflang': 'en-US',\n 'inherited': True,\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#'}],\n 'online_access_flag': True,\n 'orbit_calculated_spatial_domains': [{'orbit_number': '34851'}],\n 'original_format': 'ECHO10',\n 'polygons': [['-61.876564 79.848251 -65.236069 83.180344 -63.400402 90.778 '\n '-60.217258 86.840034 -61.876564 79.848251']],\n 'producer_granule_id': 'S1A_EW_GRDM_1SDH_20201018T140000_20201018T140100_034851_041027_2777',\n 'time_end': '2020-10-18T14:01:00.000Z',\n 'time_start': '2020-10-18T14:00:00.000Z',\n 'title': 'S1A_EW_GRDM_1SDH_20201018T140000_20201018T140100_034851_041027_2777-METADATA_GRD_MD',\n 'updated': '2020-10-20T06:51:16.000Z'},\n {'browse_flag': False,\n 'collection_concept_id': 'C1214472336-ASF',\n 'coordinate_system': 'GEODETIC',\n 'data_center': 'ASF',\n 'dataset_id': 'SENTINEL-1A_DUAL_POL_METADATA_GRD_MEDIUM_RES',\n 'day_night_flag': 'UNSPECIFIED',\n 'granule_size': '0.05633258819580078',\n 'id': 'G1954798927-ASF',\n 'links': [{'href': 'https://datapool.asf.alaska.edu/METADATA_GRD_MD/SA/S1A_EW_GRDM_1SDH_20201018T140100_20201018T140148_034851_041027_C95D.iso.xml',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',\n 'title': 'This link provides direct download access to the '\n 'granule.'},\n {'href': 'www.asf.alaska.edu/sar-data-sets/sentinel-1',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#',\n 'title': 'ASF DAAC Sentinel-1 data set landing page (VIEW RELATED '\n 'INFORMATION)'},\n {'href': 'www.asf.alaska.edu/sar-information/sentinel-1-documents-tools',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#',\n 'title': 'ASF DAAC Sentinel-1 User Guide and Technical '\n 'Documentation (VIEW RELATED INFORMATION)'},\n {'href': 'https://vertex.daac.asf.alaska.edu/',\n 'hreflang': 'en-US',\n 'inherited': True,\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#'}],\n 'online_access_flag': True,\n 'orbit_calculated_spatial_domains': [{'orbit_number': '34851'}],\n 'original_format': 'ECHO10',\n 'polygons': [['-59.113174 77.623962 -61.876228 79.847961 -60.217464 '\n '86.837784 -57.570774 84.174744 -59.113174 77.623962']],\n 'producer_granule_id': 'S1A_EW_GRDM_1SDH_20201018T140100_20201018T140148_034851_041027_C95D',\n 'time_end': '2020-10-18T14:01:48.000Z',\n 'time_start': '2020-10-18T14:01:00.000Z',\n 'title': 'S1A_EW_GRDM_1SDH_20201018T140100_20201018T140148_034851_041027_C95D-METADATA_GRD_MD',\n 'updated': '2020-10-20T06:46:01.000Z'}]" }, { - "objectID": "external/harmony_subsetting.html", - "href": "external/harmony_subsetting.html", - "title": "Data Subsetting and Transformation Services in the Cloud", + "objectID": "external/nasa_earthdata_authentication.html", + "href": "external/nasa_earthdata_authentication.html", + "title": "Authentication for NASA Earthdata", "section": "", "text": "imported on: 2023-05-30\nThis notebook was originally developed for the 2021 Cloud Hackathon, and has been updated with most current approaches." }, { - "objectID": "external/harmony_subsetting.html#using-the-harmony-py-library-to-access-customized-data-from-nasa-earthdata", - "href": "external/harmony_subsetting.html#using-the-harmony-py-library-to-access-customized-data-from-nasa-earthdata", - "title": "Data Subsetting and Transformation Services in the Cloud", - "section": "Using the Harmony-Py library to access customized data from NASA Earthdata", - "text": "Using the Harmony-Py library to access customized data from NASA Earthdata\n\n\nTiming:\n\nExercise: 40 minutes" + "objectID": "external/nasa_earthdata_authentication.html#summary", + "href": "external/nasa_earthdata_authentication.html#summary", + "title": "Authentication for NASA Earthdata", + "section": "Summary", + "text": "Summary\nThis notebook creates a hidden .netrc file (_netrc for Window OS) with Earthdata login credentials in your home directory. This file is needed to access NASA Earthdata assets from a scripting environment like Python.\n\nEarthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\nAuthentication via netrc File\nYou will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. An example of the required content is below.\nmachine urs.earthdata.nasa.gov\nlogin <USERNAME>\npassword <PASSWORD>\n<USERNAME> and <PASSWORD> would be replaced by your actual Earthdata Login username and password respectively." }, { - "objectID": "external/harmony_subsetting.html#summary", - "href": "external/harmony_subsetting.html#summary", - "title": "Data Subsetting and Transformation Services in the Cloud", - "section": "Summary", - "text": "Summary\nWe have already explored direct access to the NASA EOSDIS archive in the cloud via the Amazon Simple Storage Service (S3) by using the Common Metadata Repository (CMR) to search for granule locations. In addition to directly accessing the files archived and distributed by each of the NASA DAACs, many datasets also support services that allow us to customize the data via subsetting, reformatting, reprojection, and other transformations.\nThis tutorial demonstrates how to find, request, and use customized data from a new ecosystem of services operating within the NASA Earthdata Cloud: NASA Harmony.\n\nBenefits\nBut first, why use this option when we’ve already learned how to access data directly from the NASA Earthdata Cloud?\n\nConsistent access patterns to EOSDIS holdings make cross-data center data access easier\nData reduction services allow us to request only the data we want, in the format and projection we want\nAnalysis Ready Data and cloud access will help reduce time-to-science\nCommunity Development helps reduce the barriers for re-use of code and sharing of domain knowledge\n\n\n\n\nData file filtering and subsetting\n\n\nSee more on the Earthdata Harmony landing page, including documentation on the Harmony API itself.\n\n\nObjectives\n\nConceptualize the data transformation service types and offerings provided by NASA Earthdata, including Harmony.\nPractice skills learned from the introductory CMR tutorial to discover what access and service options exist for a given data set, as well as variable metadata.\nUtilize the Harmony-py library to request subsetted MODIS L2 Sea Surface Temperature data over the Gulf of Mexico.\nRead Harmony subsetted outputs directly into xarray. ___" + "objectID": "external/nasa_earthdata_authentication.html#import-required-packages", + "href": "external/nasa_earthdata_authentication.html#import-required-packages", + "title": "Authentication for NASA Earthdata", + "section": "Import Required Packages", + "text": "Import Required Packages\n\nfrom netrc import netrc\nfrom subprocess import Popen\nfrom platform import system\nfrom getpass import getpass\nimport os\n\nThe code below will:\n\ncheck what operating system (OS) is being used to determine which netrc file to check for/create (.netrc or _netrc)\ncheck if you have an netrc file, and if so, varify if those credentials are for the Earthdata endpoint\ncreate a netrc file if a netrc file is not present.\n\n\nurs = 'urs.earthdata.nasa.gov' # Earthdata URL endpoint for authentication\nprompts = ['Enter NASA Earthdata Login Username: ',\n 'Enter NASA Earthdata Login Password: ']\n\n# Determine the OS (Windows machines usually use an '_netrc' file)\nnetrc_name = \"_netrc\" if system()==\"Windows\" else \".netrc\"\n\n# Determine if netrc file exists, and if so, if it includes NASA Earthdata Login Credentials\ntry:\n netrcDir = os.path.expanduser(f\"~/{netrc_name}\")\n netrc(netrcDir).authenticators(urs)[0]\n\n# Below, create a netrc file and prompt user for NASA Earthdata Login Username and Password\nexcept FileNotFoundError:\n homeDir = os.path.expanduser(\"~\")\n Popen('touch {0}{2} | echo machine {1} >> {0}{2}'.format(homeDir + os.sep, urs, netrc_name), shell=True)\n Popen('echo login {} >> {}{}'.format(getpass(prompt=prompts[0]), homeDir + os.sep, netrc_name), shell=True)\n Popen('echo \\'password {} \\'>> {}{}'.format(getpass(prompt=prompts[1]), homeDir + os.sep, netrc_name), shell=True)\n # Set restrictive permissions\n Popen('chmod 0600 {0}{1}'.format(homeDir + os.sep, netrc_name), shell=True)\n\n # Determine OS and edit netrc file if it exists but is not set up for NASA Earthdata Login\nexcept TypeError:\n homeDir = os.path.expanduser(\"~\")\n Popen('echo machine {1} >> {0}{2}'.format(homeDir + os.sep, urs, netrc_name), shell=True)\n Popen('echo login {} >> {}{}'.format(getpass(prompt=prompts[0]), homeDir + os.sep, netrc_name), shell=True)\n Popen('echo \\'password {} \\'>> {}{}'.format(getpass(prompt=prompts[1]), homeDir + os.sep, netrc_name), shell=True)\n\n\nSee if the file was created\nIf the file was created, we’ll see a .netrc file (_netrc for Window OS) in the list printed below. To view the contents from a Jupyter environment, click File on the top toolbar, select Open from Path…, type .netrc, and click Open. The .netrc file will open within the text editor.\n\n!!! Beware, your password will be visible if the .netrc file is opened in the text editor.\n\n\n!ls -al ~/" }, { - "objectID": "external/harmony_subsetting.html#import-packages", - "href": "external/harmony_subsetting.html#import-packages", - "title": "Data Subsetting and Transformation Services in the Cloud", - "section": "Import Packages", - "text": "Import Packages\n\nfrom harmony import BBox, Client, Collection, Request, LinkType\nfrom harmony.config import Environment\nimport requests\nfrom pprint import pprint\nimport datetime as dt\nimport s3fs\nimport xarray as xr" + "objectID": "external/xarray.html", + "href": "external/xarray.html", + "title": "Introduction to xarray", + "section": "", + "text": "imported on: 2023-05-30\nThis notebook was originally developed for the 2021 Cloud Hackathon, and has been updated with most current approaches." }, { - "objectID": "external/harmony_subsetting.html#discover-service-options-for-a-given-data-set", - "href": "external/harmony_subsetting.html#discover-service-options-for-a-given-data-set", - "title": "Data Subsetting and Transformation Services in the Cloud", - "section": "Discover service options for a given data set", - "text": "Discover service options for a given data set\n\nFirst, what do we mean by a “service”?\nIn the context of NASA Earthdata, we are usually referring to a service as any data transformation or customization process that packages or delivers data in a way that makes it easier to work with compared to how the data are natively archived at NASA EOSDIS. Basic customization options may include: * Subsetting (cropping) the data by: * Variable * Spatial boundary, * Temporal range * Reformatting * For example: From NetCDF-4 to Cloud Optimized GeoTIFF * Reprojection and/or Resampling * For example: From Sinusoidal to Polar Stereographic * Mosaicking * Aggregating\nA few main types or pathways for services that are commonly supported across the NASA DAACs include: * NASA Global Imagery Browse Service * Web services providing imagery, much of which is updated daily, to broaden accessibility of NASA EOSDIS data to the media and public. * Web Map Tile Service (WMTS) * Tiled Web Map Service (TWMS) * Web Map Service (WMS) * Keyhole Markup Language (KML) * Geospatial Data Abstraction Library (GDAL) * OPeNDAP * The Open-source Project for a Network Data Access Protocol is a NASA community standard DAP that provides a simple way to access and work with data over the internet. OPeNDAP’s client/server software allows us to subset and reformat data using an internet browser, command line interface, and other applications. * Harmony * In the most basic sense, Harmony is an Application Programming Interface, or API, allowing us to request customization options described above, which are then processed and returned as file outputs. Harmony helps to reduce pre-processing steps so we can spend less time preparing the data, and more time doing science.\nNote: These service offerings are unique to each NASA EOSDIS dataset.\nWhy is this?\nDue to varying levels of service, cloud migration status, and unique characteristics of the datasets themselves, not all service options are provided for all datasets. Therefore it is important to first explore a given dataset’s metadata to discover what service options are provided.\nLet’s utilize the CMR API skills we learned on Day 1 to inspect service metadata:\n\nurl = 'https://cmr.earthdata.nasa.gov/search'\n\nWe want to search by collection to inspect the access and service options that exist:\n\ncollection_url = f'{url}/{\"collections\"}'\n\nIn the CMR introduction tutorial, we explored cloud-hosted collections from different DAAC providers, and identified the CMR concept-id for a given data set id (also referred to as a short_name).\nHere we are jumping ahead and already know the concept_id we are interested in, by browsing cloud-hosted datasets from PO.DAAC in Earthdata Search: https://search.earthdata.nasa.gov/portal/podaac-cloud/search.\nWe are going to focus on MODIS_A-JPL-L2P-v2019.0: GHRSST Level 2P Global Sea Surface Skin Temperature from the Moderate Resolution Imaging Spectroradiometer (MODIS) on the NASA Aqua satellite (GDS2). Let’s first save this as a variable that we can use later on once we request data from Harmony.\n\nshort_name= 'MODIS_A-JPL-L2P-v2019.0'\nconcept_id = 'C1940473819-POCLOUD'\n\nWe will view the top-level metadata for this collection to see what additional service and variable metadata exist.\n\nresponse = requests.get(collection_url, \n params={\n 'concept_id': concept_id,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\nresponse = response.json()\n\nPrint the response:\n\npprint(response)\n\n{'feed': {'entry': [{'archive_center': 'NASA/JPL/PODAAC',\n 'associations': {'services': ['S1962070864-POCLOUD',\n 'S2004184019-POCLOUD'],\n 'tools': ['TL2108419875-POCLOUD',\n 'TL2092786348-POCLOUD'],\n 'variables': ['V1997812737-POCLOUD',\n 'V1997812697-POCLOUD',\n 'V2112014688-POCLOUD',\n 'V1997812756-POCLOUD',\n 'V1997812688-POCLOUD',\n 'V1997812670-POCLOUD',\n 'V1997812724-POCLOUD',\n 'V2112014684-POCLOUD',\n 'V1997812701-POCLOUD',\n 'V1997812681-POCLOUD',\n 'V2112014686-POCLOUD',\n 'V1997812663-POCLOUD',\n 'V1997812676-POCLOUD',\n 'V1997812744-POCLOUD',\n 'V1997812714-POCLOUD']},\n 'boxes': ['-90 -180 90 180'],\n 'browse_flag': True,\n 'collection_data_type': 'SCIENCE_QUALITY',\n 'coordinate_system': 'CARTESIAN',\n 'data_center': 'POCLOUD',\n 'dataset_id': 'GHRSST Level 2P Global Sea Surface Skin '\n 'Temperature from the Moderate Resolution '\n 'Imaging Spectroradiometer (MODIS) on the '\n 'NASA Aqua satellite (GDS2)',\n 'has_formats': True,\n 'has_spatial_subsetting': True,\n 'has_temporal_subsetting': True,\n 'has_transforms': False,\n 'has_variables': True,\n 'id': 'C1940473819-POCLOUD',\n 'links': [{'href': 'https://podaac.jpl.nasa.gov/Podaac/thumbnails/MODIS_A-JPL-L2P-v2019.0.jpg',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/browse#'},\n {'href': 'https://github.com/podaac/data-readers',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://podaac-tools.jpl.nasa.gov/drive/files/OceanTemperature/ghrsst/docs/GDS20r5.pdf',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://ghrsst.jpl.nasa.gov',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://oceancolor.gsfc.nasa.gov/atbd/sst/flag/',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://oceancolor.gsfc.nasa.gov/reprocessing/r2019/sst/',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://oceancolor.gsfc.nasa.gov/atbd/sst4/',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://modis.gsfc.nasa.gov/data/atbd/atbd_mod25.pdf',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://oceancolor.gsfc.nasa.gov/atbd/sst/',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'http://www.ghrsst.org',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://podaac.jpl.nasa.gov/forum/viewforum.php?f=18&sid=e2d67e5a01815fc6e39fcd2087ed8bc8',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://podaac.jpl.nasa.gov/CitingPODAAC',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://cmr.earthdata.nasa.gov/virtual-directory/collections/C1940473819-POCLOUD',\n 'hreflang': 'en-US',\n 'length': '75.0MB',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#'},\n {'href': 'https://github.com/podaac/tutorials/blob/master/notebooks/MODIS_L2P_SST_DataCube.ipynb',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://search.earthdata.nasa.gov/search/granules?p=C1940473819-POCLOUD',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#'}],\n 'online_access_flag': True,\n 'orbit_parameters': {'inclination_angle': '98.1',\n 'number_of_orbits': '1.0',\n 'period': '98.4',\n 'swath_width': '2330.0'},\n 'organizations': ['NASA/JPL/PODAAC'],\n 'original_format': 'UMM_JSON',\n 'platforms': ['Aqua'],\n 'processing_level_id': '2',\n 'service_features': {'esi': {'has_formats': False,\n 'has_spatial_subsetting': False,\n 'has_temporal_subsetting': False,\n 'has_transforms': False,\n 'has_variables': False},\n 'harmony': {'has_formats': True,\n 'has_spatial_subsetting': True,\n 'has_temporal_subsetting': True,\n 'has_transforms': False,\n 'has_variables': True},\n 'opendap': {'has_formats': True,\n 'has_spatial_subsetting': True,\n 'has_temporal_subsetting': True,\n 'has_transforms': False,\n 'has_variables': True}},\n 'short_name': 'MODIS_A-JPL-L2P-v2019.0',\n 'summary': 'NASA produces skin sea surface temperature '\n '(SST) products from the Infrared (IR) '\n 'channels of the Moderate-resolution Imaging '\n 'Spectroradiometer (MODIS) onboard the Aqua '\n 'satellite. Aqua was launched by NASA on May '\n '4, 2002, into a sun synchronous, polar orbit '\n 'with a daylight ascending node at 1:30 pm, '\n 'formation flying in the A-train with other '\n 'Earth Observation Satellites (EOS), to study '\n 'the global dynamics of the Earth atmosphere, '\n 'land and oceans. MODIS captures data in 36 '\n 'spectral bands at a variety of spatial '\n 'resolutions. Two SST products can be present '\n 'in these files. The first is a skin SST '\n 'produced for both day and night (NSST) '\n 'observations, derived from the long wave IR '\n '11 and 12 micron wavelength channels, using a '\n 'modified nonlinear SST algorithm intended to '\n 'provide continuity of SST derived from '\n 'heritage and current NASA sensors. At night, '\n 'a second SST product is generated using the '\n 'mid-infrared 3.95 and 4.05 micron wavelength '\n 'channels which are unique to MODIS; the SST '\n 'derived from these measurements is identified '\n 'as SST4. The SST4 product has lower '\n 'uncertainty, but due to sun glint can only be '\n 'used at night. MODIS L2P SST data have a 1 km '\n 'spatial resolution at nadir and are stored in '\n '288 five minute granules per day. Full global '\n 'coverage is obtained every two days, with '\n 'coverage poleward of 32.3 degree being '\n 'complete each day. The production of MODIS '\n 'L2P SST files is part of the Group for High '\n 'Resolution Sea Surface Temperature (GHRSST) '\n 'project and is a joint collaboration between '\n 'the NASA Jet Propulsion Laboratory (JPL), the '\n 'NASA Ocean Biology Processing Group (OBPG), '\n 'and the Rosenstiel School of Marine and '\n 'Atmospheric Science (RSMAS). Researchers at '\n 'RSMAS are responsible for SST algorithm '\n 'development, error statistics and quality '\n 'flagging, while the OBPG, as the NASA ground '\n 'data system, is responsible for the '\n 'production of daily MODIS ocean products. JPL '\n 'acquires MODIS ocean granules from the OBPG '\n 'and reformats them to the GHRSST L2P netCDF '\n 'specification with complete metadata and '\n 'ancillary variables, and distributes the data '\n 'as the official Physical Oceanography Data '\n 'Archive (PO.DAAC) for SST. The R2019.0 '\n 'supersedes the previous R2014.0 datasets '\n 'which can be found at '\n 'https://doi.org/10.5067/GHMDA-2PJ02',\n 'time_start': '2002-07-04T00:00:00.000Z',\n 'title': 'GHRSST Level 2P Global Sea Surface Skin '\n 'Temperature from the Moderate Resolution '\n 'Imaging Spectroradiometer (MODIS) on the NASA '\n 'Aqua satellite (GDS2)',\n 'updated': '2019-12-02T22:59:24.849Z',\n 'version_id': '2019.0'}],\n 'id': 'https://cmr.earthdata.nasa.gov:443/search/collections.json?concept_id=C1940473819-POCLOUD',\n 'title': 'ECHO dataset metadata',\n 'updated': '2021-11-19T17:44:29.026Z'}}\n\n\n\n\nWhat do each of these service values mean?\n\nAssociations\n\nCMR is a large web of interconnected metadata “schemas”, including Collections, Granules, Services, Tools, and Variables. In this case, this collection is associated with two unique services, two tools, and several unique variables.\n\nTags\n\nThere are also tags that describe what service options exist at a high-level. In this case, we see that this dataset supports the ability to reformat, subset by space and time, as well as by variable. This is used in web applications like Earthdata Search to surface those customization options more readily.\n\nService Features\n\nIn this case, we see three separate “features” listed here: esi, Harmony, and OPeNDAP.\n\n\nWe will dig into more details on what Harmony offers for this dataset.\nFirst, we need to isolate the services returned for this dataset:\n\nservices = response['feed']['entry'][0]['associations']['services']\nprint(services)\n\n['S1962070864-POCLOUD', 'S2004184019-POCLOUD']\n\n\n\nservice_url = \"https://cmr.earthdata.nasa.gov/search/services\"\n\nInspect the first service returned. Now we’re going to search the services endpoint to view that individual service’s metadata, like we did with our dataset above. This time, we’re explicitly setting the format of the response to umm-json in the Accept Header in order to see detailed metadata about the service.\n\nservice_response = requests.get(service_url, \n params={\n 'concept_id': services[0],\n },\n headers={\n 'Accept': 'application/vnd.nasa.cmr.umm_results+json'\n }\n )\nservice_response = service_response.json()\n\nDetails about the service metadata record include the service options provided by the “backend” processor connected to Harmony, in this case the PODAAC Level 2 Cloud Subsetter:\n\npprint(service_response)\n\n{'hits': 1,\n 'items': [{'meta': {'concept-id': 'S1962070864-POCLOUD',\n 'concept-type': 'service',\n 'deleted': False,\n 'format': 'application/vnd.nasa.cmr.umm+json',\n 'native-id': 'POCLOUD_podaac_l2_cloud_subsetter',\n 'provider-id': 'POCLOUD',\n 'revision-date': '2021-11-02T22:57:03.597Z',\n 'revision-id': 19,\n 'user-id': 'podaaccloud'},\n 'umm': {'AccessConstraints': 'None',\n 'Description': 'Endpoint for subsetting L2 Subsetter via '\n 'Harmony',\n 'LongName': 'PODAAC Level 2 Cloud Subsetter',\n 'MetadataSpecification': {'Name': 'UMM-S',\n 'URL': 'https://cdn.earthdata.nasa.gov/umm/service/v1.4',\n 'Version': '1.4'},\n 'Name': 'PODAAC L2 Cloud Subsetter',\n 'OperationMetadata': [{'OperationName': 'SPATIAL_SUBSETTING'},\n {'OperationName': 'VARIABLE_SUBSETTING'},\n {'OperationName': 'TEMPORAL_SUBSETTING'}],\n 'ServiceKeywords': [{'ServiceCategory': 'EARTH SCIENCE '\n 'SERVICES',\n 'ServiceTerm': 'SUBSETTING/SUPERSETTING',\n 'ServiceTopic': 'DATA MANAGEMENT/DATA '\n 'HANDLING'}],\n 'ServiceOptions': {'Subset': {'SpatialSubset': {'BoundingBox': {'AllowMultipleValues': False}},\n 'TemporalSubset': {'AllowMultipleValues': False},\n 'VariableSubset': {'AllowMultipleValues': True}},\n 'SupportedReformattings': [{'SupportedInputFormat': 'HDF5',\n 'SupportedOutputFormats': ['NETCDF-4']},\n {'SupportedInputFormat': 'NETCDF-4',\n 'SupportedOutputFormats': ['NETCDF-4']}]},\n 'ServiceOrganizations': [{'LongName': 'Physical '\n 'Oceanography '\n 'Distributed Active '\n 'Archive Center, Jet '\n 'Propulsion '\n 'Laboratory, NASA',\n 'Roles': ['ORIGINATOR'],\n 'ShortName': 'NASA/JPL/PODAAC'}],\n 'Type': 'Harmony',\n 'URL': {'Description': 'PROJECT HOME PAGE',\n 'URLValue': 'https://harmony.earthdata.nasa.gov'},\n 'Version': '1.1.0'}}],\n 'took': 9}\n\n\n\n\nDiscover all datasets that support Harmony services\nInstead of searching for services on a known dataset of interest, we may want to discovery all available datasets that are supported for a given service. We can utilize GraphQL, which is a way for us to efficiently gain information across service and collection metadata so that we can print out all supported Harmony datasets. First, we need to specify a query string. Here we are asking to query all collections with service type “Harmony”, and to provide details on the service options attached to those services:\n\nquery = \"\"\"query {\n collections(limit: 2000, serviceType: \"Harmony\") {\n count\n items {\n shortName\n conceptId\n services {\n count\n items {\n name\n supportedReformattings\n supportedInputProjections\n supportedOutputProjections\n serviceOptions\n }\n }\n variables {\n count\n }\n }\n }\n}\"\"\"\n\nThis utilizes a different API endpoint to query CMR metdata using GraphQL. Here we set up another request, passing our query string above:\n\ngraphql_url = 'https://graphql.earthdata.nasa.gov/api'\n\ngraphql_response = requests.get(graphql_url,\n params={\"query\": query},\n headers={\n 'Accept': 'application/json',\n }\n )\n\nA json response is returned that provides all collections with Harmony-supported services. We can then extract just the collectionshortName, conceptID, and the service names supported for each collection:\n\nservices = graphql_response.json()['data']['collections']['items']\n\nfor service in services:\n print(service['shortName'], \",\", service['conceptId'])\n for i in range(len(service['services']['items'])):\n print(\"Services:\", service['services']['items'][i]['name'])\n\nECCO_L4_ATM_STATE_05DEG_DAILY_V4R4 , C1990404801-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_ATM_STATE_LLC0090GRID_DAILY_V4R4 , C1991543823-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_ATM_STATE_05DEG_MONTHLY_V4R4 , C1990404814-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_ATM_STATE_LLC0090GRID_MONTHLY_V4R4 , C1991543805-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_BOLUS_STREAMFUNCTION_LLC0090GRID_DAILY_V4R4 , C1991543818-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_BOLUS_STREAMFUNCTION_LLC0090GRID_MONTHLY_V4R4 , C1991543733-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_BOLUS_05DEG_DAILY_V4R4 , C1990404807-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_BOLUS_LLC0090GRID_DAILY_V4R4 , C1991543824-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_BOLUS_05DEG_MONTHLY_V4R4 , C1990404805-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_BOLUS_LLC0090GRID_MONTHLY_V4R4 , C1991543745-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_GEOMETRY_05DEG_V4R4 , C2013583732-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_GEOMETRY_LLC0090GRID_V4R4 , C2013557893-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_GMAP_TIME_SERIES_SNAPSHOT_V4R4 , C1991543729-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_GMAP_TIME_SERIES_SNAPSHOT_V4R4B , C2133160276-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_GMSL_TIME_SERIES_DAILY_V4R4 , C1991543819-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_GMSL_TIME_SERIES_MONTHLY_V4R4 , C1991543742-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_MIX_COEFFS_05DEG_V4R4 , C2013584708-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_MIX_COEFFS_LLC0090GRID_V4R4 , C2013583906-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_FRESH_FLUX_05DEG_DAILY_V4R4 , C1990404818-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_FRESH_FLUX_LLC0090GRID_DAILY_V4R4 , C1991543820-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_FRESH_FLUX_05DEG_MONTHLY_V4R4 , C1990404792-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_FRESH_FLUX_LLC0090GRID_MONTHLY_V4R4 , C1991543803-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_HEAT_FLUX_05DEG_DAILY_V4R4 , C1990404788-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_HEAT_FLUX_LLC0090GRID_DAILY_V4R4 , C1991543712-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_HEAT_FLUX_05DEG_MONTHLY_V4R4 , C1990404812-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_HEAT_FLUX_LLC0090GRID_MONTHLY_V4R4 , C1991543811-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_STRESS_05DEG_DAILY_V4R4 , C1990404808-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_STRESS_LLC0090GRID_DAILY_V4R4 , C1991543704-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_STRESS_05DEG_MONTHLY_V4R4 , C1990404796-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_STRESS_LLC0090GRID_MONTHLY_V4R4 , C1991543760-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_05DEG_DAILY_V4R4 , C1990404797-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_05DEG_DAILY_V4R4B , C2129192243-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_LLC0090GRID_DAILY_V4R4 , C1991543737-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_LLC0090GRID_DAILY_V4R4B , C2129195053-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_05DEG_MONTHLY_V4R4 , C1990404791-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_05DEG_MONTHLY_V4R4B , C2129193421-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_LLC0090GRID_MONTHLY_V4R4 , C1991543806-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_LLC0090GRID_MONTHLY_V4R4B , C2129197196-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_LLC0090GRID_SNAPSHOT_V4R4 , C1991543804-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_DENS_STRAT_PRESS_05DEG_DAILY_V4R4 , C1990404793-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_DENS_STRAT_PRESS_LLC0090GRID_DAILY_V4R4 , C1991543727-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_DENS_STRAT_PRESS_05DEG_MONTHLY_V4R4 , C1990404798-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_DENS_STRAT_PRESS_LLC0090GRID_MONTHLY_V4R4 , C1991543735-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_MIXED_LAYER_DEPTH_05DEG_DAILY_V4R4 , C1990404810-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_MIXED_LAYER_DEPTH_LLC0090GRID_DAILY_V4R4 , C1991543734-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_MIXED_LAYER_DEPTH_05DEG_MONTHLY_V4R4 , C1990404819-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_MIXED_LAYER_DEPTH_LLC0090GRID_MONTHLY_V4R4 , C1991543741-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_TEMP_SALINITY_05DEG_DAILY_V4R4 , C1990404821-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_TEMP_SALINITY_LLC0090GRID_DAILY_V4R4 , C1991543736-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_TEMP_SALINITY_05DEG_MONTHLY_V4R4 , C1990404795-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_TEMP_SALINITY_LLC0090GRID_MONTHLY_V4R4 , C1991543728-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_TEMP_SALINITY_LLC0090GRID_SNAPSHOT_V4R4 , C1991543757-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_MOMENTUM_TEND_LLC0090GRID_DAILY_V4R4 , C1991543726-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_MOMENTUM_TEND_LLC0090GRID_MONTHLY_V4R4 , C1991543702-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_TEMPERATURE_FLUX_LLC0090GRID_DAILY_V4R4 , C1991543812-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_TEMPERATURE_FLUX_LLC0090GRID_MONTHLY_V4R4 , C1991543740-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_SALINITY_FLUX_LLC0090GRID_DAILY_V4R4 , C1991543814-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_SALINITY_FLUX_LLC0090GRID_MONTHLY_V4R4 , C1991543752-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_VOLUME_FLUX_LLC0090GRID_DAILY_V4R4 , C1991543699-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_VOLUME_FLUX_LLC0090GRID_MONTHLY_V4R4 , C1991543739-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_VEL_05DEG_DAILY_V4R4 , C1990404811-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_VEL_LLC0090GRID_DAILY_V4R4 , C1991543808-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_VEL_05DEG_MONTHLY_V4R4 , C1990404823-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_VEL_LLC0090GRID_MONTHLY_V4R4 , C1991543732-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SBO_CORE_TIME_SERIES_SNAPSHOT_V4R4 , C1991543766-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SBO_CORE_TIME_SERIES_SNAPSHOT_V4R4B , C2133162585-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_05DEG_DAILY_V4R4 , C1990404813-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_05DEG_DAILY_V4R4B , C2129181904-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_LLC0090GRID_DAILY_V4R4 , C1991543744-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_LLC0090GRID_DAILY_V4R4B , C2129186341-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_05DEG_MONTHLY_V4R4 , C1990404799-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_05DEG_MONTHLY_V4R4B , C2129189405-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_LLC0090GRID_MONTHLY_V4R4 , C1991543813-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_LLC0090GRID_MONTHLY_V4R4B , C2129189870-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_LLC0090GRID_SNAPSHOT_V4R4 , C1991543817-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_CONC_THICKNESS_05DEG_DAILY_V4R4 , C1990404815-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_CONC_THICKNESS_LLC0090GRID_DAILY_V4R4 , C1991543763-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_CONC_THICKNESS_05DEG_MONTHLY_V4R4 , C1990404820-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_CONC_THICKNESS_LLC0090GRID_MONTHLY_V4R4 , C1991543764-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_CONC_THICKNESS_LLC0090GRID_SNAPSHOT_V4R4 , C1991543821-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_HORIZ_VOLUME_FLUX_LLC0090GRID_DAILY_V4R4 , C1991543731-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_HORIZ_VOLUME_FLUX_LLC0090GRID_MONTHLY_V4R4 , C1991543724-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_SALT_PLUME_FLUX_LLC0090GRID_DAILY_V4R4 , C1991543807-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_SALT_PLUME_FLUX_LLC0090GRID_MONTHLY_V4R4 , C1991543730-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_VELOCITY_05DEG_DAILY_V4R4 , C1990404817-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_VELOCITY_LLC0090GRID_DAILY_V4R4 , C1991543765-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_VELOCITY_05DEG_MONTHLY_V4R4 , C1990404790-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_VELOCITY_LLC0090GRID_MONTHLY_V4R4 , C1991543700-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_VELOCITY_LLC0090GRID_SNAPSHOT_V4R4 , C1991543768-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nVIIRS_NPP-NAVO-L2P-v1.0 , C1996881807-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nVIIRS_NPP-NAVO-L2P-v3.0 , C1996881636-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nMSG03-OSPO-L2P-v1.0 , C2036878029-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nVIIRS_NPP-JPL-L2P-v2016.2 , C1996881456-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nMODIS_A-JPL-L2P-v2019.0 , C1940473819-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nMODIS_A-JPL-L2P-v2019.0 , C1693233348-PODAAC\nServices: PODAAC L2 Cloud Subsetter\nMODIS_T-JPL-L2P-v2019.0 , C1940475563-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nMODIS_T-JPL-L2P-v2019.0 , C1693233387-PODAAC\nServices: PODAAC L2 Cloud Subsetter\nTMI-REMSS-L2P-v4 , C2036879048-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nVIIRS_NPP-OSPO-L2P-v2.61 , C1996880725-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nVIIRS_N20-OSPO-L2P-v2.61 , C1996880450-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nAVHRR_SST_METOP_B-OSISAF-L2P-v1.0 , C2036880717-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nJASON-1_L2_OST_GPR_E , C1684065156-PODAAC\nServices: PODAAC L2 Cloud Subsetter\nJASON-1_L2_OST_GPN_E , C1684065153-PODAAC\nServices: PODAAC L2 Cloud Subsetter\nJASON-1_L2_OST_GPS_E , C1684065158-PODAAC\nServices: PODAAC L2 Cloud Subsetter\nCYGNSS_NOAA_L2_SWSP_25KM_V1.1 , C2036882072-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nJASON_CS_S6A_L2_AMR_RAD_NRT , C1968979997-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nJASON_CS_S6A_L2_AMR_RAD_STC , C1968979762-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nJASON_CS_S6A_L2_ALT_LR_STD_OST_NRT_F , C1968979597-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nJASON_CS_S6A_L2_ALT_LR_RED_OST_NRT_F , C1968980576-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nJASON_CS_S6A_L2_ALT_LR_STD_OST_STC_F , C1968980609-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nJASON_CS_S6A_L2_ALT_LR_RED_OST_STC_F , C1968979561-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter" + "objectID": "external/xarray.html#why-do-we-need-xarray", + "href": "external/xarray.html#why-do-we-need-xarray", + "title": "Introduction to xarray", + "section": "Why do we need xarray?", + "text": "Why do we need xarray?\nAs Geoscientists, we often work with time series of data with two or more dimensions: a time series of calibrated, orthorectified satellite images; two-dimensional grids of surface air temperature from an atmospheric reanalysis; or three-dimensional (level, x, y) cubes of ocean salinity from an ocean model. These data are often provided in GeoTIFF, NetCDF or HDF format with rich and useful metadata that we want to retain, or even use in our analysis. Common analyses include calculating means, standard deviations and anomalies over time or one or more spatial dimensions (e.g. zonal means). Model output often includes multiple variables that you want to apply similar analyses to.\n\n\n\nA schematic of multi-dimensional data\n\n\nThe schematic above shows a typical data structure for multi-dimensional data. There are two data cubes, one for temperature and one for precipitation. Common coordinate variables, in this case latitude, longitude and time are associated with each variable. Each variable, including coordinate variables, will have a set of attributes: name, units, missing value, etc. The file containing the data may also have attributes: source of the data, model name coordinate reference system if the data are projected. Writing code using low-level packages such as netcdf4 and numpy to read the data, then perform analysis, and write the results to file is time consuming and prone to errors." }, { - "objectID": "external/harmony_subsetting.html#discover-variable-names", - "href": "external/harmony_subsetting.html#discover-variable-names", - "title": "Data Subsetting and Transformation Services in the Cloud", - "section": "Discover variable names", - "text": "Discover variable names\nJust like services, a dataset may also be associated to metadata on their individual data variables, which can be used as input if we wish to subset by one or more variables of interest.\n\nvariables = response['feed']['entry'][0]['associations']['variables']\nprint(variables)\n\n['V1997812737-POCLOUD', 'V1997812697-POCLOUD', 'V2112014688-POCLOUD', 'V1997812756-POCLOUD', 'V1997812688-POCLOUD', 'V1997812670-POCLOUD', 'V1997812724-POCLOUD', 'V2112014684-POCLOUD', 'V1997812701-POCLOUD', 'V1997812681-POCLOUD', 'V2112014686-POCLOUD', 'V1997812663-POCLOUD', 'V1997812676-POCLOUD', 'V1997812744-POCLOUD', 'V1997812714-POCLOUD']\n\n\nSeveral variable records are returned. Again, like we did for services, we’ll search the variables endpoint to view an individual variable’s metadata, and we’ll print out the list of variables for our dataset.\n\nvar_url = \"https://cmr.earthdata.nasa.gov/search/variables\"\n\n\nvar_response = requests.get(var_url, \n params={\n 'concept_id': variables[0],\n },\n headers={\n 'Accept': 'application/vnd.nasa.cmr.umm_results+json'\n }\n )\nvar_response = var_response.json()\n\n\npprint(var_response)\n\n{'hits': 1,\n 'items': [{'associations': {'collections': [{'concept-id': 'C1940473819-POCLOUD'}]},\n 'meta': {'concept-id': 'V1997812737-POCLOUD',\n 'concept-type': 'variable',\n 'deleted': False,\n 'format': 'application/vnd.nasa.cmr.umm+json',\n 'native-id': 'MODIS_A-JPL-L2P-v2019.0-sses_standard_deviation_4um',\n 'provider-id': 'POCLOUD',\n 'revision-date': '2021-10-19T02:26:51.560Z',\n 'revision-id': 6,\n 'user-id': 'jmcnelis'},\n 'umm': {'DataType': 'byte',\n 'Definition': 'mid-IR SST standard deviation error; non '\n 'L2P core field; signed byte array: WARNING '\n 'Some applications are unable to properly '\n 'handle signed byte values. If values are '\n 'encountered > 127, please subtract 256 from '\n 'this reported value',\n 'Dimensions': [{'Name': 'time',\n 'Size': 1,\n 'Type': 'TIME_DIMENSION'},\n {'Name': 'nj',\n 'Size': 2030,\n 'Type': 'ALONG_TRACK_DIMENSION'},\n {'Name': 'ni',\n 'Size': 1354,\n 'Type': 'CROSS_TRACK_DIMENSION'}],\n 'FillValues': [{'Type': 'SCIENCE_FILLVALUE',\n 'Value': -128}],\n 'LongName': 'SSES standard deviation error based on '\n 'proximity confidence flags',\n 'Name': 'sses_standard_deviation_4um',\n 'Offset': 10.0,\n 'Scale': 0.07874016,\n 'Sets': [{'Index': 1,\n 'Name': 'sses_standard_deviation_4um',\n 'Size': 1,\n 'Type': 'General'}],\n 'Units': 'kelvin',\n 'ValidRanges': [{'Max': 127, 'Min': -127}],\n 'VariableType': 'ANCILLARY_VARIABLE'}}],\n 'took': 9}\n\n\nNext, print out a simple list of all associated variable names by looping the same variable response we submitted above, this time for each variable:\n\nvar_list = []\nfor i in range(len(variables)):\n var_response = requests.get(var_url, \n params={\n 'concept_id': variables[i],\n },\n headers={\n 'Accept': 'application/vnd.nasa.cmr.umm_results+json'\n }\n )\n var_response = var_response.json()\n var_list.append(var_response['items'][0]['umm']['Name'])\n\n\npprint(var_list)\n\n['sses_standard_deviation_4um',\n 'l2p_flags',\n 'time',\n 'dt_analysis',\n 'sses_standard_deviation',\n 'sst_dtime',\n 'sses_bias_4um',\n 'lat',\n 'sea_surface_temperature_4um',\n 'sses_bias',\n 'lon',\n 'sea_surface_temperature',\n 'quality_level',\n 'wind_speed',\n 'quality_level_4um']" + "objectID": "external/xarray.html#what-is-xarray", + "href": "external/xarray.html#what-is-xarray", + "title": "Introduction to xarray", + "section": "What is xarray", + "text": "What is xarray\nxarray is an open-source project and python package to work with labelled multi-dimensional arrays. It is leverages numpy, pandas, matplotlib and dask to build Dataset and DataArray objects with built-in methods to subset, analyze, interpolate, and plot multi-dimensional data. It makes working with multi-dimensional data cubes efficient and fun. It will change your life for the better. You’ll be more attractive, more interesting, and better equiped to take on lifes challenges." }, { - "objectID": "external/harmony_subsetting.html#using-harmony-py-to-subset-data", - "href": "external/harmony_subsetting.html#using-harmony-py-to-subset-data", - "title": "Data Subsetting and Transformation Services in the Cloud", - "section": "Using Harmony-Py to subset data", - "text": "Using Harmony-Py to subset data\nHarmony-Py provides a pip installable Python alternative to directly using Harmony’s RESTful API to make it easier to request data and service options, especially when interacting within a Python Jupyter Notebook environment.\nThe next steps are adopted from the introduction tutorial notebook provided in the Harmony-Py library:\n\nCreate Harmony Client object\nFirst, we need to create a Harmony Client, which is what we will interact with to submit and inspect a data request to Harmony, as well as to retrieve results.\nWhen creating the Client, we need to provide Earthdata Login credentials, which are required to access data from NASA EOSDIS. This basic line below assumes that we have a .netrc available.\n\nharmony_client = Client()\n\n\n\nHurricane Ida snapshot\nUsing NASA Worldview, we can first explore SST during a tropical storm event; in this case, we can overlay L2 and L4 SST variables against true color imagery to observe Hurricane Ida in August 2021. Although this is a small sample set, this use case could be expanded to explore how SST responds during the Atlantic hurricane over the next several months. The same data that we are requesting below using Harmony-py can also be requested using NASA Earthdata Search\n\n\n\nHurrican Ida snapshot - Worldview\n\n\n\n\nCreate Harmony Request\nThe following are common request parameters:\n\ncollection: Required parameter. This is the NASA EOSDIS collection, or data product. There are two options for inputting a collection of interest:\n\nProvide a concept ID (e.g. C1940473819-POCLOUD)\nData product short name (e.g. MODIS_A-JPL-L2P-v2019.0).\n\nspatial: Bounding box spatial constraints on the data. The Harmony Bbox class accepts spatial coordinates as decimal degrees in w, s, e, n order, where longitude = -180, 180 and latitude = -90, 90.\ntemporal: Date/time constraints on the data. The example below demonstrates temporal start and end ranges using the python datetime library.\n\nAs we identified above, only subsetting options are available for this dataset. If other service options such as reformatting are available for a given dataset, these can also be specified using Harmony-py: See the documentation for details on how to construct these parameters.\n\nrequest = Request(\n collection=Collection(id=short_name),\n spatial=BBox(-97.77667,21.20806,-83.05197,30.16605),\n temporal={\n 'start': dt.datetime(2021, 8, 20),\n 'stop': dt.datetime(2021, 8, 21),\n },\n)\n\n\n\nCheck Request validity\nBefore submitting a Harmony Request, we can test your request to see if it’s valid and how to fix it if not. In particular, request.is_valid will check to ensure that the spatial BBox bounds and temporal ranges are entered correctly.\n\nrequest.is_valid()\n\nTrue\n\n\n\n\nSubmit request\nNow that the request is created, we can now submit it to Harmony using the Harmony Client object. A job id is returned, which is a unique identifier that represents the submitted request.\n\njob_id = harmony_client.submit(request)\njob_id\n\n'e36f6916-a7d9-4d82-a5fa-924d48d464ef'\n\n\n\n\nCheck request status\nWe can check on the progress of a processing job with status(). This method blocks while communicating with the server but returns quickly.\n\nharmony_client.status(job_id)\n\n{'status': 'running',\n 'message': 'There were 2 collections that matched the provided short name MODIS_A-JPL-L2P-v2019.0. See https://cmr.earthdata.nasa.gov/concepts/C1940473819-POCLOUD for details on the selected collection. The version ID for the selected collection is 2019.0. To use a different collection submit a new request specifying the desired CMR concept ID instead of the collection short name.',\n 'progress': 0,\n 'created_at': datetime.datetime(2021, 11, 19, 17, 44, 40, 768000, tzinfo=tzlocal()),\n 'updated_at': datetime.datetime(2021, 11, 19, 17, 44, 40, 768000, tzinfo=tzlocal()),\n 'request': 'https://harmony.earthdata.nasa.gov/MODIS_A-JPL-L2P-v2019.0/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(21.20806%3A30.16605)&subset=lon(-97.77667%3A-83.05197)&subset=time(%222021-08-20T00%3A00%3A00%22%3A%222021-08-21T00%3A00%3A00%22)',\n 'num_input_granules': 6}\n\n\nDepending on the size of the request, it may be helpful to wait until the request has completed processing before the remainder of the code is executed. The wait_for_processing() method will block subsequent lines of code while optionally showing a progress bar.\n\nharmony_client.wait_for_processing(job_id, show_progress=True)\n\n [ Processing: 100% ] |###################################################| [|]\n\n\n\n\nView Harmony job response and output URLs\nOnce the data request has finished processing, we can view details on the job that was submitted to Harmony, including the API call to Harmony, and informational messages on the request if available.\nresult_json() calls wait_for_processing() and returns the complete job in JSON format once processing is complete.\n\ndata = harmony_client.result_json(job_id)\npprint(data)\n\n{'createdAt': '2021-11-19T17:44:40.768Z',\n 'jobID': 'e36f6916-a7d9-4d82-a5fa-924d48d464ef',\n 'links': [{'href': 'https://harmony.earthdata.nasa.gov/stac/e36f6916-a7d9-4d82-a5fa-924d48d464ef/',\n 'rel': 'stac-catalog-json',\n 'title': 'STAC catalog',\n 'type': 'application/json'},\n {'bbox': [-83.612, 1.103, -58.391, 22.005],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820062501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T06:29:58.000Z',\n 'start': '2021-08-20T06:25:01.000Z'},\n 'title': '20210820062501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-97.8, 21.2, -83.1, 30.2],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820080001-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T08:04:58.000Z',\n 'start': '2021-08-20T08:00:01.000Z'},\n 'title': '20210820080001-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-128.335, 29.006, -95.854, 49.87],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820093501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T09:39:49.000Z',\n 'start': '2021-08-20T09:35:01.000Z'},\n 'title': '20210820093501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-96.8, 21.2, -83.1, 27.7],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820185501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T18:59:58.000Z',\n 'start': '2021-08-20T18:55:01.000Z'},\n 'title': '20210820185501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-97.8, 25.3, -83.1, 30.2],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820190001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T19:04:58.000Z',\n 'start': '2021-08-20T19:00:01.000Z'},\n 'title': '20210820190001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-97.8, 21.2, -97.3, 24.6],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820203501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T20:39:58.000Z',\n 'start': '2021-08-20T20:35:01.000Z'},\n 'title': '20210820203501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'href': 'https://harmony.earthdata.nasa.gov/jobs/e36f6916-a7d9-4d82-a5fa-924d48d464ef?linktype=https&page=1&limit=2000',\n 'rel': 'self',\n 'title': 'The current page',\n 'type': 'application/json'}],\n 'message': 'There were 2 collections that matched the provided short name '\n 'MODIS_A-JPL-L2P-v2019.0. See '\n 'https://cmr.earthdata.nasa.gov/concepts/C1940473819-POCLOUD for '\n 'details on the selected collection. The version ID for the '\n 'selected collection is 2019.0. To use a different collection '\n 'submit a new request specifying the desired CMR concept ID '\n 'instead of the collection short name.',\n 'numInputGranules': 6,\n 'progress': 100,\n 'request': 'https://harmony.earthdata.nasa.gov/MODIS_A-JPL-L2P-v2019.0/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(21.20806%3A30.16605)&subset=lon(-97.77667%3A-83.05197)&subset=time(%222021-08-20T00%3A00%3A00%22%3A%222021-08-21T00%3A00%3A00%22)',\n 'status': 'successful',\n 'updatedAt': '2021-11-19T17:45:17.148Z',\n 'username': 'amy.steiker'}\n\n\n\n\nDirect cloud access\nNote that the remainder of this tutorial will only succeed when running this notebook within the AWS us-west-2 region.\nHarmony data outputs can be accessed within the cloud using the s3 URLs and AWS credentials provided in the Harmony job response.\n\nRetrieve list of output URLs.\nThe result_urls() method calls wait_for_processing() and returns a list of the processed data URLs once processing is complete. You may optionally show the progress bar as shown below.\n\nresults = harmony_client.result_urls(job_id, link_type=LinkType.s3)\nurls = list(results)\npprint(urls)\n\n['s3://harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820062501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820080001-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0_subsetted.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820093501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820185501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820190001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820203501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4']\n\n\nWe can see that the first file returned does not include the _subsetted suffix, which indicates that a blank file was returned, as no data values were located within our subsetted region. We’ll select the second URL in the list to bring into xarray below.\n\nurl = urls[1]\nurl\n\n's3://harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820080001-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0_subsetted.nc4'\n\n\n\n\nAWS credential retrieval\nUsing aws_credentials you can retrieve the credentials needed to access the Harmony s3 staging bucket and its contents.\n\ncreds = harmony_client.aws_credentials()\n\n\n\n\nOpen staged files with s3fs and xarray\nWe use the AWS s3fs package to create a file system that can then be read by xarray:\n\ns3_fs = s3fs.S3FileSystem(\n key=creds['aws_access_key_id'],\n secret=creds['aws_secret_access_key'],\n token=creds['aws_session_token'],\n client_kwargs={'region_name':'us-west-2'},\n)\n\nNow that we have our s3 file system set, including our declared credentials, we’ll use that to open the url, and read in the file through xarray. This extra step is needed because xarray cannot open the S3 location directly. Instead, the S3 file object is passed to xarray, in order to then open the dataset.\n\nf = s3_fs.open(url, mode='rb')\nds = xr.open_dataset(f)\nds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (nj: 1232, ni: 1132, time: 1)\nCoordinates:\n lat (nj, ni, time) float32 ...\n lon (nj, ni, time) float32 ...\n * time (time) datetime64[ns] 2021-08-20T08:00:01\nDimensions without coordinates: nj, ni\nData variables:\n sea_surface_temperature (time, nj, ni) float32 ...\n sst_dtime (time, nj, ni) timedelta64[ns] ...\n quality_level (time, nj, ni) float32 ...\n sses_bias (time, nj, ni) float32 ...\n sses_standard_deviation (time, nj, ni) float32 ...\n l2p_flags (time, nj, ni) int16 ...\n sea_surface_temperature_4um (time, nj, ni) float32 ...\n quality_level_4um (time, nj, ni) float32 ...\n sses_bias_4um (time, nj, ni) float32 ...\n sses_standard_deviation_4um (time, nj, ni) float32 ...\n wind_speed (time, nj, ni) float32 ...\n dt_analysis (time, nj, ni) float32 ...\nAttributes: (12/50)\n Conventions: CF-1.7, ACDD-1.3\n title: MODIS Aqua L2P SST\n summary: Sea surface temperature retrievals produced a...\n references: GHRSST Data Processing Specification v2r5\n institution: NASA/JPL/OBPG/RSMAS\n history: MODIS L2P created at JPL PO.DAAC\\n2021-11-19 ...\n ... ...\n processing_level: L2P\n cdm_data_type: swath\n startDirection: Descending\n endDirection: Descending\n day_night_flag: Night\n history_json: [{\"date_time\": \"2021-11-19T17:44:56.897126+00...xarray.DatasetDimensions:nj: 1232ni: 1132time: 1Coordinates: (3)lat(nj, ni, time)float32...long_name :latitudestandard_name :latitudeunits :degrees_northvalid_min :[-90.]valid_max :[90.]comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[1394624 values with dtype=float32]lon(nj, ni, time)float32...long_name :longitudestandard_name :longitudeunits :degrees_eastvalid_min :[-180.]valid_max :[180.]comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[1394624 values with dtype=float32]time(time)datetime64[ns]2021-08-20T08:00:01long_name :reference time of sst filestandard_name :timecomment :time of first sensor observationcoverage_content_type :coordinatearray(['2021-08-20T08:00:01.000000000'], dtype='datetime64[ns]')Data variables: (12)sea_surface_temperature(time, nj, ni)float32...long_name :sea surface temperaturestandard_name :sea_surface_skin_temperatureunits :kelvinvalid_min :[-1000]valid_max :[10000]comment :sea surface temperature from thermal IR (11 um) channelssource :NASA and University of Miamicoverage_content_type :physicalMeasurement[1394624 values with dtype=float32]sst_dtime(time, nj, ni)timedelta64[ns]...long_name :time difference from reference timevalid_min :[-32767]valid_max :[32767]comment :time plus sst_dtime gives seconds after 00:00:00 UTC January 1, 1981coverage_content_type :referenceInformation[1394624 values with dtype=timedelta64[ns]]quality_level(time, nj, ni)float32...long_name :quality level of SST pixelvalid_min :[0]valid_max :[5]comment :thermal IR SST proximity confidence value; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valueflag_values :[0 1 2 3 4 5]flag_meanings :no_data bad_data worst_quality low_quality acceptable_quality best_qualitycoverage_content_type :qualityInformation[1394624 values with dtype=float32]sses_bias(time, nj, ni)float32...long_name :SSES bias error based on proximity confidence flagsunits :kelvinvalid_min :[-127]valid_max :[127]comment :thermal IR SST bias error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[1394624 values with dtype=float32]sses_standard_deviation(time, nj, ni)float32...long_name :SSES standard deviation error based on proximity confidence flagsunits :kelvinvalid_min :[-127]valid_max :[127]comment :thermal IR SST standard deviation error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[1394624 values with dtype=float32]l2p_flags(time, nj, ni)int16...long_name :L2P flagsvalid_min :[0]valid_max :[16]comment :These flags can be used to further filter data variablesflag_meanings :microwave land ice lake riverflag_masks :[ 1 2 4 8 16]coverage_content_type :qualityInformation[1394624 values with dtype=int16]sea_surface_temperature_4um(time, nj, ni)float32...long_name :sea surface temperatureunits :kelvinvalid_min :[-1000]valid_max :[10000]comment :sea surface temperature from mid-IR (4 um) channels; non L2P core fieldcoverage_content_type :physicalMeasurement[1394624 values with dtype=float32]quality_level_4um(time, nj, ni)float32...long_name :quality level of SST pixelvalid_min :[0]valid_max :[5]comment :mid-IR SST proximity confidence value; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valueflag_values :[0 1 2 3 4 5]flag_meanings :no_data bad_data worst_quality low_quality acceptable_quality best_qualitycoverage_content_type :qualityInformation[1394624 values with dtype=float32]sses_bias_4um(time, nj, ni)float32...long_name :SSES bias error based on proximity confidence flagsunits :kelvinvalid_min :[-127]valid_max :[127]comment :mid-IR SST bias error; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[1394624 values with dtype=float32]sses_standard_deviation_4um(time, nj, ni)float32...long_name :SSES standard deviation error based on proximity confidence flagsunits :kelvinvalid_min :[-127]valid_max :[127]comment :mid-IR SST standard deviation error; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[1394624 values with dtype=float32]wind_speed(time, nj, ni)float32...long_name :10m wind speedstandard_name :wind_speedunits :m s-1valid_min :[-127]valid_max :[127]comment :Wind at 10 meters above the sea surfacesource :TBD. Placeholder. Currently emptygrid_mapping :TBDtime_offset :[2.]height :10 mcoverage_content_type :auxiliaryInformation[1394624 values with dtype=float32]dt_analysis(time, nj, ni)float32...long_name :deviation from SST reference climatologyunits :kelvinvalid_min :[-127]valid_max :[127]comment :TBDsource :TBD. Placeholder. Currently emptycoverage_content_type :auxiliaryInformation[1394624 values with dtype=float32]Attributes: (50)Conventions :CF-1.7, ACDD-1.3title :MODIS Aqua L2P SSTsummary :Sea surface temperature retrievals produced at the NASA OBPG for the MODIS Aqua sensor. These have been reformatted to GHRSST GDS specifications by the JPL PO.DAACreferences :GHRSST Data Processing Specification v2r5institution :NASA/JPL/OBPG/RSMAShistory :MODIS L2P created at JPL PO.DAAC\n2021-11-19 17:44:56.897087 l2ss-py v1.1.0 (bbox=[[-97.77667, -83.05197], [21.20806, 30.16605]] cut=True)comment :L2P Core without DT analysis or other ancillary fields; Night, Start Node:Descending, End Node:Descending; WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value; Quicklooklicense :GHRSST and PO.DAAC protocol allow data use as free and open.id :MODIS_A-JPL-L2P-v2019.0naming_authority :org.ghrsstproduct_version :2019.0uuid :f6e1f61d-c4a4-4c17-8354-0c15e12d688bgds_version_id :2.0netcdf_version_id :4.1date_created :20210820T110619Zfile_quality_level :[3]spatial_resolution :1kmstart_time :20210820T080001Ztime_coverage_start :20210820T080001Zstop_time :20210820T080458Ztime_coverage_end :20210820T080458Znorthernmost_latitude :[35.9455]southernmost_latitude :[14.8953]easternmost_longitude :[-78.2345]westernmost_longitude :[-105.765]source :MODIS sea surface temperature observations for the OBPGplatform :Aquasensor :MODISmetadata_link :http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MODIS_A-JPL-L2P-v2019.0keywords :Oceans > Ocean Temperature > Sea Surface Temperaturekeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordsstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventiongeospatial_lat_units :degrees_northgeospatial_lat_resolution :[0.01]geospatial_lon_units :degrees_eastgeospatial_lon_resolution :[0.01]acknowledgment :The MODIS L2P sea surface temperature data are sponsored by NASAcreator_name :Ed Armstrong, JPL PO.DAACcreator_email :edward.m.armstrong@jpl.nasa.govcreator_url :http://podaac.jpl.nasa.govproject :Group for High Resolution Sea Surface Temperaturepublisher_name :The GHRSST Project Officepublisher_url :http://www.ghrsst.orgpublisher_email :ghrsst-po@nceo.ac.ukprocessing_level :L2Pcdm_data_type :swathstartDirection :DescendingendDirection :Descendingday_night_flag :Nighthistory_json :[{\"date_time\": \"2021-11-19T17:44:56.897126+00:00\", \"derived_from\": \"https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20210820080001-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\", \"program\": \"l2ss-py\", \"version\": \"1.1.0\", \"parameters\": \"bbox=[[-97.77667, -83.05197], [21.20806, 30.16605]] cut=True\", \"program_ref\": \"https://cmr.earthdata.nasa.gov:443/search/concepts/S1962070864-POCLOUD\", \"$schema\": \"https://harmony.earthdata.nasa.gov/schemas/history/0.1.0/history-v0.1.0.json\"}]\n\n\n\n\nPlot the data\nUse the xarray built in plotting function to create a simple plot along the x and y dimensions of the dataset:\n\nds.sea_surface_temperature.plot() ;" + "objectID": "external/xarray.html#what-you-will-learn-from-this-tutorial", + "href": "external/xarray.html#what-you-will-learn-from-this-tutorial", + "title": "Introduction to xarray", + "section": "What you will learn from this tutorial", + "text": "What you will learn from this tutorial\nIn this tutorial you will learn how to:\n\nload a netcdf file into xarray\ninterrogate the Dataset and understand the difference between DataArray and Dataset\nsubset a Dataset\ncalculate annual and monthly mean fields\ncalculate a time series of zonal means\nplot these results\n\nAs always, we’ll start by importing xarray. We’ll follow convention by giving the module the shortname xr\n\nimport xarray as xr\nxr.set_options(keep_attrs=True)\n\nI’m going to use one of xarray’s tutorial datasets. In this case, air temperature from the NCEP reanalysis. I’ll assign the result of the open_dataset to ds. I may change this to access a dataset directly\n\nds = xr.tutorial.open_dataset(\"air_temperature\")\n\nAs we are in an interactive environment, we can just type ds to see what we have.\n\nds\n\nFirst thing to notice is that ds is an xarray.Dataset object. It has dimensions, lat, lon, and time. It also has coordinate variables with the same names as these dimensions. These coordinate variables are 1-dimensional. This is a NetCDF convention. The Dataset contains one data variable, air. This has dimensions (time, lat, lon).\nClicking on the document icon reveals attributes for each variable. Clicking on the disk icon reveals a representation of the data.\nEach of the data and coordinate variables can be accessed and examined using the variable name as a key.\n\nds.air\n\n\nds['air']\n\nThese are xarray.DataArray objects. This is the basic building block for xarray.\nVariables can also be accessed as attributes of ds.\n\nds.time\n\nA major difference between accessing a variable as an attribute versus using a key is that the attribute is read-only but the key method can be used to update the variable. For example, if I want to convert the units of air from Kelvin to degrees Celsius.\n\nds['air'] = ds.air - 273.15\n\nThis approach can also be used to add new variables\n\nds['air_kelvin'] = ds.air + 273.15\n\nIt is helpful to update attributes such as units, this saves time, confusion and mistakes, especially when you save the dataset.\n\nds['air'].attrs['units'] = 'degC'\n\n\nds" }, { - "objectID": "external/harmony_subsetting.html#resources", - "href": "external/harmony_subsetting.html#resources", - "title": "Data Subsetting and Transformation Services in the Cloud", - "section": "Resources", - "text": "Resources\n\nIn-depth exploration of the MODIS_A-JPL-L2P-v2019.0 data set, co-locating in-situ and remote sensing data: https://github.com/podaac/tutorials/blob/master/notebooks/SWOT-EA-2021/Colocate_satellite_insitu_ocean.ipynb\nHarmony-Py library introduction tutorial: https://github.com/nasa/harmony-py/blob/main/examples/intro_tutorial.ipynb" + "objectID": "external/xarray.html#subsetting-and-indexing", + "href": "external/xarray.html#subsetting-and-indexing", + "title": "Introduction to xarray", + "section": "Subsetting and Indexing", + "text": "Subsetting and Indexing\nSubsetting and indexing methods depend on whether you are working with a Dataset or DataArray. A DataArray can be accessed using positional indexing just like a numpy array. To access the temperature field for the first time step, you do the following.\n\nds['air'][0,:,:]\n\nNote this returns a DataArray with coordinates but not attributes.\nHowever, the real power is being able to access variables using coordinate variables. I can get the same subset using the following. (It’s also more explicit about what is being selected and robust in case I modify the DataArray and expect the same output.)\n\nds['air'].sel(time='2013-01-01').time\n\n\nds.air.sel(time='2013-01-01')\n\nI can also do slices. I’ll extract temperatures for the state of Colorado. The bounding box for the state is [-109 E, -102 E, 37 N, 41 N].\nIn the code below, pay attention to both the order of the coordinates and the range of values. The first value of the lat coordinate variable is 41 N, the second value is 37 N. Unfortunately, xarray expects slices of coordinates to be in the same order as the coordinates. Note lon is 0 to 360 not -180 to 180, and I let python calculate it for me within the slice.\n\nds.air.sel(lat=slice(41.,37.), lon=slice(360-109,360-102))\n\nWhat if we want temperature for a point, for example Denver, CO (39.72510678889283 N, -104.98785545855408 E). xarray can handle this! If we just want data from the nearest grid point, we can use sel and specify the method as “nearest”.\n\ndenver_lat, denver_lon = 39.72510678889283, -104.98785545855408\n\n\nds.air.sel(lat=denver_lat, lon=360+denver_lon, method='nearest')\n\nIf we want to interpolate, we can use interp(). In this case I use linear or bilinear interpolation.\ninterp() can also be used to resample data to a new grid and even reproject data\n\nds.air.interp(lat=denver_lat, lon=360+denver_lon, method='linear')\n\nsel() and interp() can also be used on Dataset objects.\n\nds.sel(lat=slice(41.,37.), lon=slice(360-109,360-102))\n\n\nds.interp(lat=denver_lat, lon=360+denver_lon, method='linear')" }, { - "objectID": "external/sentinel-6_opendap_access_gridding.html", - "href": "external/sentinel-6_opendap_access_gridding.html", - "title": "Sentinel-6 MF L2 Altimetry Data Access (OPeNDAP) & Gridding", - "section": "", - "text": "imported on: 2023-05-30\n\nThis notebook was originally developed for the 2021 Cloud Hackathon, and has been updated with most current approaches.\n\n\nThe original source for this document is https://nasa-openscapes.github.io/2021-Cloud-Hackathon/tutorials/06_S6_OPeNDAP_Access_Gridding.html\n\n\n06. Sentinel-6 MF L2 Altimetry Data Access (OPeNDAP) & Gridding\nIn this tutorial you will learn…\n\nabout level 2 radar altimetry data from the Sentinel-6 Michael Freilich mission;\nhow to efficiently download variable subsets using OPeNDAP;\nhow to grid the along-track altimetry observations produced by S6 at level 2.;\n\n\nAbout Ocean Surface Topography (OST)\n\nThe primary contribution of satellite altimetry to satellite oceanography has been to:\n\nImprove the knowledge of ocean tides and develop global tide models.\nMonitor the variation of global mean sea level and its relationship to changes in ocean mass and heat content.\nMap the general circulation variability of the ocean, including the ocean mesoscale, over decades and in near real-time using multi-satellite altimetric sampling.\n\n\n\nAbout Sentinel-6 MF\nhttps://search.earthdata.nasa.gov/search?fpj=Sentinel-6\n\nhttps://podaac.jpl.nasa.gov/Sentinel-6\n\nMission Characteristics\nSemi-major axis: 7,714.43 km\nEccentricity: 0.000095\nInclination: 66.04°\nArgument of periapsis: 90.0°\nMean anomaly: 253.13°\nReference altitude: 1,336 km\nNodal period: 6,745.72 sec\nRepeat period: 9.9156 days\nNumber of revolutions within a cycle: 127\nNumber of passes within a cycle: 254\nEquatorial cross track separation: 315 km\nGround track control band: +1 km\nAcute angle at Equator crossings: 39.5°\nGround track speed: 5.8 km/s\n\n\n\nRequirements\nThis workflow was developed using Python 3.9 (and tested against versions 3.7, 3.8).\n\nimport os\nimport tqdm\nimport numpy as np\nimport xarray as xr\nimport matplotlib.pyplot as plt\nfrom concurrent.futures import ThreadPoolExecutor\nfrom pyresample.kd_tree import resample_gauss\nimport pyresample as pr\n\n\n\nDataset\nhttps://podaac.jpl.nasa.gov/dataset/JASON_CS_S6A_L2_ALT_LR_RED_OST_NRT_F\nThis example operates on Level 2 Low Resolution Altimetry from Sentinel-6 Michael Freilich (the Near Real Time Reduced distribution). It is most easily identified by its collection ShortName, given below with the more cryptic concept-id, it’s unique identifier in the CMR.\n\nShortName = 'JASON_CS_S6A_L2_ALT_LR_RED_OST_NRT_F'\nconcept_id = 'C1968980576-POCLOUD'\n\n\n\ncycle = 25\n\nurl = f\"https://cmr.earthdata.nasa.gov/search/granules.csv?ShortName={ShortName}&cycle={cycle}&page_size=200\"\n\nprint(url)\n\nhttps://cmr.earthdata.nasa.gov/search/granules.csv?ShortName=JASON_CS_S6A_L2_ALT_LR_RED_OST_NRT_F&cycle=25&page_size=200\n\n\ncurl --silent --output\ncat\ntail --lines\ncut --delimiter --fields\n\n!curl --silent --output \"results.csv\" \"$url\"\n\nfiles = !cat results.csv | tail --lines=+2 | cut --delimiter=',' --fields=5 | cut --delimiter='/' --fields=6\n\nprint(files.s.replace(\" \", \"\\n\"))\n\nS6A_P4_2__LR_RED__NR_025_001_20210713T162644_20210713T182234_F02.nc\nS6A_P4_2__LR_RED__NR_025_003_20210713T182234_20210713T201839_F02.nc\nS6A_P4_2__LR_RED__NR_025_006_20210713T201839_20210713T215450_F02.nc\nS6A_P4_2__LR_RED__NR_025_007_20210713T215450_20210713T234732_F02.nc\nS6A_P4_2__LR_RED__NR_025_009_20210713T234732_20210714T014224_F02.nc\nS6A_P4_2__LR_RED__NR_025_011_20210714T014224_20210714T033812_F02.nc\nS6A_P4_2__LR_RED__NR_025_013_20210714T033812_20210714T053356_F02.nc\nS6A_P4_2__LR_RED__NR_025_015_20210714T053357_20210714T072934_F02.nc\nS6A_P4_2__LR_RED__NR_025_017_20210714T072934_20210714T090919_F02.nc\nS6A_P4_2__LR_RED__NR_025_019_20210714T090919_20210714T110146_F02.nc\nS6A_P4_2__LR_RED__NR_025_021_20210714T110146_20210714T125702_F02.nc\nS6A_P4_2__LR_RED__NR_025_023_20210714T125702_20210714T145316_F02.nc\nS6A_P4_2__LR_RED__NR_025_025_20210714T145317_20210714T164922_F02.nc\nS6A_P4_2__LR_RED__NR_025_027_20210714T164922_20210714T184510_F02.nc\nS6A_P4_2__LR_RED__NR_025_029_20210714T184510_20210714T204143_F02.nc\nS6A_P4_2__LR_RED__NR_025_032_20210714T204143_20210714T221611_F02.nc\nS6A_P4_2__LR_RED__NR_025_033_20210714T221611_20210715T000941_F02.nc\nS6A_P4_2__LR_RED__NR_025_035_20210715T000941_20210715T020456_F02.nc\nS6A_P4_2__LR_RED__NR_025_037_20210715T020456_20210715T040047_F02.nc\nS6A_P4_2__LR_RED__NR_025_039_20210715T040047_20210715T055630_F02.nc\nS6A_P4_2__LR_RED__NR_025_041_20210715T055630_20210715T075208_F02.nc\nS6A_P4_2__LR_RED__NR_025_043_20210715T075208_20210715T093037_F02.nc\nS6A_P4_2__LR_RED__NR_025_045_20210715T093037_20210715T112356_F02.nc\nS6A_P4_2__LR_RED__NR_025_047_20210715T112356_20210715T131944_F02.nc\nS6A_P4_2__LR_RED__NR_025_049_20210715T131944_20210715T151600_F02.nc\nS6A_P4_2__LR_RED__NR_025_051_20210715T151602_20210715T165851_F02.nc\nS6A_P4_2__LR_RED__NR_025_053_20210715T171228_20210715T190748_F02.nc\nS6A_P4_2__LR_RED__NR_025_056_20210715T190748_20210715T204627_F02.nc\nS6A_P4_2__LR_RED__NR_025_057_20210715T204627_20210715T223758_F02.nc\nS6A_P4_2__LR_RED__NR_025_059_20210715T223758_20210716T003159_F02.nc\nS6A_P4_2__LR_RED__NR_025_061_20210716T003159_20210716T022732_F02.nc\nS6A_P4_2__LR_RED__NR_025_063_20210716T022732_20210716T042333_F02.nc\nS6A_P4_2__LR_RED__NR_025_065_20210716T042333_20210716T061901_F02.nc\nS6A_P4_2__LR_RED__NR_025_067_20210716T061901_20210716T081446_F02.nc\nS6A_P4_2__LR_RED__NR_025_070_20210716T081446_20210716T095203_F02.nc\nS6A_P4_2__LR_RED__NR_025_071_20210716T095203_20210716T114624_F02.nc\nS6A_P4_2__LR_RED__NR_025_073_20210716T114624_20210716T134228_F02.nc\nS6A_P4_2__LR_RED__NR_025_075_20210716T134228_20210716T153841_F02.nc\nS6A_P4_2__LR_RED__NR_025_077_20210716T153841_20210716T173433_F02.nc\nS6A_P4_2__LR_RED__NR_025_079_20210716T173433_20210716T193033_F02.nc\nS6A_P4_2__LR_RED__NR_025_082_20210716T193033_20210716T210718_F02.nc\nS6A_P4_2__LR_RED__NR_025_083_20210716T210718_20210716T225942_F02.nc\nS6A_P4_2__LR_RED__NR_025_085_20210716T225942_20210717T005425_F02.nc\nS6A_P4_2__LR_RED__NR_025_087_20210717T005425_20210717T025012_F02.nc\nS6A_P4_2__LR_RED__NR_025_089_20210717T025012_20210717T044557_F02.nc\nS6A_P4_2__LR_RED__NR_025_091_20210717T044557_20210717T064133_F02.nc\nS6A_P4_2__LR_RED__NR_025_093_20210717T064133_20210717T082134_F02.nc\nS6A_P4_2__LR_RED__NR_025_095_20210717T082134_20210717T101352_F02.nc\nS6A_P4_2__LR_RED__NR_025_097_20210717T101352_20210717T120859_F02.nc\nS6A_P4_2__LR_RED__NR_025_099_20210717T120859_20210717T140513_F02.nc\nS6A_P4_2__LR_RED__NR_025_101_20210717T140513_20210717T160120_F02.nc\nS6A_P4_2__LR_RED__NR_025_103_20210717T160120_20210717T175708_F02.nc\nS6A_P4_2__LR_RED__NR_025_105_20210717T175708_20210717T195329_F02.nc\nS6A_P4_2__LR_RED__NR_025_108_20210717T195329_20210717T212832_F02.nc\nS6A_P4_2__LR_RED__NR_025_109_20210717T212832_20210717T232147_F02.nc\nS6A_P4_2__LR_RED__NR_025_111_20210717T232147_20210718T011655_F02.nc\nS6A_P4_2__LR_RED__NR_025_113_20210718T011655_20210718T031245_F02.nc\nS6A_P4_2__LR_RED__NR_025_115_20210718T031245_20210718T050829_F02.nc\nS6A_P4_2__LR_RED__NR_025_117_20210718T050829_20210718T070406_F02.nc\nS6A_P4_2__LR_RED__NR_025_119_20210718T070406_20210718T084306_F02.nc\nS6A_P4_2__LR_RED__NR_025_121_20210718T084306_20210718T103559_F02.nc\nS6A_P4_2__LR_RED__NR_025_123_20210718T103559_20210718T123140_F02.nc\nS6A_P4_2__LR_RED__NR_025_125_20210718T123140_20210718T142756_F02.nc\nS6A_P4_2__LR_RED__NR_025_127_20210718T142756_20210718T162356_F02.nc\nS6A_P4_2__LR_RED__NR_025_129_20210718T162356_20210718T181945_F02.nc\nS6A_P4_2__LR_RED__NR_025_132_20210718T181945_20210718T195907_F02.nc\nS6A_P4_2__LR_RED__NR_025_133_20210718T195907_20210718T215014_F02.nc\nS6A_P4_2__LR_RED__NR_025_135_20210718T215014_20210718T234402_F02.nc\nS6A_P4_2__LR_RED__NR_025_137_20210718T234402_20210719T013937_F02.nc\nS6A_P4_2__LR_RED__NR_025_139_20210719T013937_20210719T033531_F02.nc\nS6A_P4_2__LR_RED__NR_025_141_20210719T033531_20210719T053101_F02.nc\nS6A_P4_2__LR_RED__NR_025_143_20210719T053101_20210719T072643_F02.nc\nS6A_P4_2__LR_RED__NR_025_146_20210719T072643_20210719T090425_F02.nc\nS6A_P4_2__LR_RED__NR_025_147_20210719T090425_20210719T105824_F02.nc\nS6A_P4_2__LR_RED__NR_025_149_20210719T105824_20210719T125424_F02.nc\nS6A_P4_2__LR_RED__NR_025_151_20210719T125424_20210719T145038_F02.nc\nS6A_P4_2__LR_RED__NR_025_153_20210719T145541_20210719T164632_F02.nc\nS6A_P4_2__LR_RED__NR_025_155_20210719T164632_20210719T184227_F02.nc\nS6A_P4_2__LR_RED__NR_025_158_20210719T184227_20210719T201949_F02.nc\nS6A_P4_2__LR_RED__NR_025_159_20210719T201949_20210719T221154_F02.nc\nS6A_P4_2__LR_RED__NR_025_161_20210719T221154_20210720T000626_F02.nc\nS6A_P4_2__LR_RED__NR_025_163_20210720T000626_20210720T020212_F02.nc\nS6A_P4_2__LR_RED__NR_025_165_20210720T020212_20210720T035756_F02.nc\nS6A_P4_2__LR_RED__NR_025_167_20210720T035756_20210720T055333_F02.nc\nS6A_P4_2__LR_RED__NR_025_169_20210720T055333_20210720T073350_F02.nc\nS6A_P4_2__LR_RED__NR_025_171_20210720T073350_20210720T092602_F02.nc\nS6A_P4_2__LR_RED__NR_025_173_20210720T092602_20210720T112057_F02.nc\nS6A_P4_2__LR_RED__NR_025_175_20210720T112057_20210720T131708_F02.nc\nS6A_P4_2__LR_RED__NR_025_177_20210720T131708_20210720T151317_F02.nc\nS6A_P4_2__LR_RED__NR_025_184_20210720T190549_20210720T204056_F02.nc\nS6A_P4_2__LR_RED__NR_025_185_20210720T204056_20210720T223355_F02.nc\nS6A_P4_2__LR_RED__NR_025_187_20210720T223355_20210721T002855_F02.nc\nS6A_P4_2__LR_RED__NR_025_189_20210721T002855_20210721T022443_F02.nc\nS6A_P4_2__LR_RED__NR_025_191_20210721T022443_20210721T042029_F02.nc\nS6A_P4_2__LR_RED__NR_025_193_20210721T042029_20210721T061605_F02.nc\nS6A_P4_2__LR_RED__NR_025_195_20210721T061605_20210721T075531_F02.nc\nS6A_P4_2__LR_RED__NR_025_197_20210721T075531_20210721T094805_F02.nc\nS6A_P4_2__LR_RED__NR_025_199_20210721T094805_20210721T114336_F02.nc\nS6A_P4_2__LR_RED__NR_025_201_20210721T114336_20210721T133952_F02.nc\nS6A_P4_2__LR_RED__NR_025_203_20210721T133952_20210721T153555_F02.nc\nS6A_P4_2__LR_RED__NR_025_205_20210721T153555_20210721T173143_F02.nc\nS6A_P4_2__LR_RED__NR_025_207_20210721T173143_20210721T191151_F02.nc\nS6A_P4_2__LR_RED__NR_025_209_20210721T191151_20210721T210223_F02.nc\nS6A_P4_2__LR_RED__NR_025_211_20210721T210223_20210721T225607_F02.nc\nS6A_P4_2__LR_RED__NR_025_213_20210721T225607_20210722T005131_F02.nc\nS6A_P4_2__LR_RED__NR_025_215_20210722T005131_20210722T024724_F02.nc\nS6A_P4_2__LR_RED__NR_025_217_20210722T024724_20210722T044301_F02.nc\nS6A_P4_2__LR_RED__NR_025_219_20210722T044301_20210722T063841_F02.nc\nS6A_P4_2__LR_RED__NR_025_221_20210722T063841_20210722T081646_F02.nc\nS6A_P4_2__LR_RED__NR_025_223_20210722T081646_20210722T101025_F02.nc\nS6A_P4_2__LR_RED__NR_025_225_20210722T101025_20210722T120619_F02.nc\nS6A_P4_2__LR_RED__NR_025_227_20210722T120619_20210722T140235_F02.nc\nS6A_P4_2__LR_RED__NR_025_229_20210722T140235_20210722T155831_F02.nc\nS6A_P4_2__LR_RED__NR_025_231_20210722T155831_20210722T175423_F02.nc\nS6A_P4_2__LR_RED__NR_025_234_20210722T175423_20210722T193222_F02.nc\nS6A_P4_2__LR_RED__NR_025_235_20210722T193222_20210722T212406_F02.nc\nS6A_P4_2__LR_RED__NR_025_237_20210722T212406_20210722T231828_F02.nc\nS6A_P4_2__LR_RED__NR_025_239_20210722T231828_20210723T011405_F02.nc\nS6A_P4_2__LR_RED__NR_025_241_20210723T011405_20210723T030955_F02.nc\nS6A_P4_2__LR_RED__NR_025_243_20210723T030955_20210723T050533_F02.nc\nS6A_P4_2__LR_RED__NR_025_245_20210723T050533_20210723T064603_F02.nc\nS6A_P4_2__LR_RED__NR_025_247_20210723T064603_20210723T083817_F02.nc\nS6A_P4_2__LR_RED__NR_025_249_20210723T083817_20210723T103256_F02.nc\nS6A_P4_2__LR_RED__NR_025_251_20210723T103256_20210723T122904_F02.nc\nS6A_P4_2__LR_RED__NR_025_253_20210723T122904_20210723T142514_F02.nc\n\n\n\nOPeNDAP\nhttps://opendap.github.io/documentation/UserGuideComprehensive.html#Constraint_Expressions (Hyrax/OPeNDAP docs)\n\ntmp = files.l[0].split('.')[0]\n\nprint(f\"https://opendap.earthdata.nasa.gov/collections/{concept_id}/granules/{tmp}.html\")\n\nhttps://opendap.earthdata.nasa.gov/collections/C1968980576-POCLOUD/granules/S6A_P4_2__LR_RED__NR_025_001_20210713T162644_20210713T182234_F02.html\n\n\n\nvariables = ['data_01_time',\n 'data_01_longitude',\n 'data_01_latitude',\n 'data_01_ku_ssha']\n\n\nv = \",\".join(variables)\n\nurls = []\nfor f in files:\n urls.append(f\"https://opendap.earthdata.nasa.gov/collections/{concept_id}/granules/{f}4?{v}\")\n\nprint(urls[0])\n\nhttps://opendap.earthdata.nasa.gov/collections/C1968980576-POCLOUD/granules/S6A_P4_2__LR_RED__NR_025_001_20210713T162644_20210713T182234_F02.nc4?data_01_time,data_01_longitude,data_01_latitude,data_01_ku_ssha\n\n\n\n\n\nDownload Subsets\nThese functions download one granule from the remote source to a local target, and will reliably manage simultaneous streaming downloads divided between multiple threads.\n\nwith python3:\nimport requests\n\ndef download(source: str, target: str):\n with requests.get(source, stream=True) as remote, open(target, 'wb') as local:\n if remote.status_code // 100 == 2: \n for chunk in remote.iter_content(chunk_size=1024):\n if chunk:\n local.write(chunk)\n\n\nwith wget:\n\ndef download(source: str):\n \n target = os.path.basename(source.split(\"?\")[0])\n \n if not os.path.isfile(target):\n !wget --quiet --continue --output-document $target $source\n \n return target\n\n\nn_workers = 12\n\nwith ThreadPoolExecutor(max_workers=n_workers) as pool:\n\n workers = pool.map(download, urls)\n \n files = list(tqdm.tqdm(workers, total=len(urls)))\n\n100%|██████████| 125/125 [01:01<00:00, 2.05it/s]\n\n\nhttps://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor\nThe source files range from 2.5MB to 3.0MB. These OPeNDAP subsets are ~100KB apiece. (anecdote: it took less than 10 minutes to download subsets for >1700 granules/files when I ran this routine for all cycles going back to 2021-06-22.)\n\n!du -sh .\n\n17M .\n\n\nhttps://www.gnu.org/software/coreutils/manual/html_node/du-invocation.html\n\n\nAggregate cycle\nSort the list of local subsets to ensure they concatenate in proper order. Call open_mfdataset on the list to open all the subsets in memory as one dataset in xarray.\n\nds = xr.open_mfdataset(sorted(files))\n\nprint(ds)\n\n<xarray.Dataset>\nDimensions: (data_01_time: 827001)\nCoordinates:\n * data_01_time (data_01_time) datetime64[ns] 2021-07-13T16:26:45 ... ...\nData variables:\n data_01_longitude (data_01_time) float64 dask.array<chunksize=(6950,), meta=np.ndarray>\n data_01_latitude (data_01_time) float64 dask.array<chunksize=(6950,), meta=np.ndarray>\n data_01_ku_ssha (data_01_time) float64 dask.array<chunksize=(6950,), meta=np.ndarray>\nAttributes: (12/63)\n Convention: CF-1.7\n institution: EUMETSAT\n references: Sentinel-6_Jason-CS ALT Generic P...\n contact: ops@eumetsat.int\n radiometer_sensor_name: AMR-C\n doris_sensor_name: DORIS\n ... ...\n xref_solid_earth_tide: S6__P4_2__SETD_AX_20151008T000000...\n xref_surface_classification: S6__P4____SURF_AX_20151008T000000...\n xref_wind_speed_alt: S6A_P4_2__WNDL_AX_20151008T000000...\n product_name: S6A_P4_2__LR______20210713T162644...\n history: 2021-07-13 18:38:07 : Creation\\n2...\n history_json: [{\"$schema\":\"https:\\/\\/harmony.ea...\n\n\nhttps://xarray.pydata.org/en/stable/generated/xarray.open_mfdataset.html\nMake a dictionary to rename variables so that the data_01_ prefix is removed from each one.\n\nnew_variable_names = list(map(lambda x: x.split(\"_\")[-1], variables))\n\nmap_variable_names = dict(zip(variables, new_variable_names))\n\nmap_variable_names\n\n{'data_01_time': 'time',\n 'data_01_longitude': 'longitude',\n 'data_01_latitude': 'latitude',\n 'data_01_ku_ssha': 'ssha'}\n\n\nhttps://docs.python.org/3/library/functions.html#map\nhttps://docs.python.org/3/library/functions.html#zip\n\nds = ds.rename(map_variable_names)\n\nprint(list(ds.variables))\n\n['longitude', 'latitude', 'ssha', 'time']\n\n\n\n\nPlot ssha variable\nhttps://xarray.pydata.org/en/stable/generated/xarray.Dataset.rename.html\n\nds.plot.scatter( y=\"latitude\",\n x=\"longitude\", \n hue=\"ssha\",\n s=1,\n vmin=-0.4,\n vmax=0.4,\n levels=9, \n cmap=\"jet\",\n aspect=2.5,\n size=9, )\n\nplt.title(f\"ssha from s6 cycle {cycle}\")\nplt.xlim( 0., 360.)\nplt.ylim(-67., 67.)\nplt.show()\n\n\n\n\n\n\n\nBorrow 0.5-Degree Grid and Mask from ECCO V4r4\n\nAcknowledgement: This approach using pyresample was shared to me by Ian Fenty, ECCO Lead.\n\nhttps://search.earthdata.nasa.gov/search/granules?p=C2013583732-POCLOUD\nECCO V4r4 products are distributed in two spatial formats. One set of collections provides the ocean state estimates on the native model grid (LLC0090) and the other provides them after interpolating to a regular grid defined in geographic coordinates with horizontal cell size of 0.5-degrees.\nIt’s distributed as its own dataset/collection containing just one file. We can simply download it from the HTTPS download endpoint – the file size is inconsequential. The next cell downloads the file into the data folder from the granule’s https endpoint.\n\necco_url = \"https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/ECCO_L4_GEOMETRY_05DEG_V4R4/GRID_GEOMETRY_ECCO_V4r4_latlon_0p50deg.nc\"\n\necco_file = download(ecco_url)\n\necco_grid = xr.open_dataset(ecco_file)\n\nprint(ecco_grid)\n\n<xarray.Dataset>\nDimensions: (Z: 50, latitude: 360, longitude: 720, nv: 2)\nCoordinates:\n * Z (Z) float32 -5.0 -15.0 -25.0 ... -5.461e+03 -5.906e+03\n * latitude (latitude) float32 -89.75 -89.25 -88.75 ... 89.25 89.75\n * longitude (longitude) float32 -179.8 -179.2 -178.8 ... 179.2 179.8\n latitude_bnds (latitude, nv) float32 ...\n longitude_bnds (longitude, nv) float32 ...\n Z_bnds (Z, nv) float32 ...\nDimensions without coordinates: nv\nData variables:\n hFacC (Z, latitude, longitude) float64 ...\n Depth (latitude, longitude) float64 ...\n area (latitude, longitude) float64 ...\n drF (Z) float32 ...\n maskC (Z, latitude, longitude) bool ...\nAttributes: (12/57)\n acknowledgement: This research was carried out by the Jet...\n author: Ian Fenty and Ou Wang\n cdm_data_type: Grid\n comment: Fields provided on a regular lat-lon gri...\n Conventions: CF-1.8, ACDD-1.3\n coordinates_comment: Note: the global 'coordinates' attribute...\n ... ...\n references: ECCO Consortium, Fukumori, I., Wang, O.,...\n source: The ECCO V4r4 state estimate was produce...\n standard_name_vocabulary: NetCDF Climate and Forecast (CF) Metadat...\n summary: This dataset provides geometric paramete...\n title: ECCO Geometry Parameters for the 0.5 deg...\n uuid: b4795c62-86e5-11eb-9c5f-f8f21e2ee3e0\n\n\nhttps://xarray.pydata.org/en/stable/generated/xarray.open_dataset.html\nSelect index 0 on the Z axis/dimension to get the depth layer at ocean surface.\n\necco_grid = ecco_grid.isel(Z=0).copy()\n\nhttps://xarray.pydata.org/en/stable/generated/xarray.DataArray.isel.html\nThe maskC variable contains a boolean mask representing the wet/dry state of the area contained in each cell of the 3d grid defined by Z and latitude and longitude. Here are the variable’s attributes:\n\nprint(ecco_grid.maskC)\n\n<xarray.DataArray 'maskC' (latitude: 360, longitude: 720)>\n[259200 values with dtype=bool]\nCoordinates:\n Z float32 -5.0\n * latitude (latitude) float32 -89.75 -89.25 -88.75 ... 88.75 89.25 89.75\n * longitude (longitude) float32 -179.8 -179.2 -178.8 ... 178.8 179.2 179.8\nAttributes:\n coverage_content_type: modelResult\n long_name: wet/dry boolean mask for grid cell\n comment: True for grid cells with nonzero open vertical fr...\n\n\nPlot the land/water mask maskC:\n\necco_grid.maskC.plot()\n\n<matplotlib.collections.QuadMesh at 0x7fed730b00a0>\n\n\n\n\n\nhttps://xarray.pydata.org/en/stable/generated/xarray.DataArray.plot.html\nDefine target grid based on the longitudes and latitudes from the ECCO grid geometry dataset. This time define the grid using two 2-dimensional arrays that give positions of all SSHA values in geographic/longitude-latitude coordinates.\n\necco_lons = ecco_grid.maskC.longitude.values\necco_lats = ecco_grid.maskC.latitude.values\n\necco_lons_2d, ecco_lats_2d = np.meshgrid(ecco_lons, ecco_lats)\n\nprint(ecco_lons_2d.shape, ecco_lats_2d.shape)\n\n(360, 720) (360, 720)\n\n\nCreate the target swath definition from the 2d arrays of lons and lats from ECCO V4r4 0.5-degree grid.\n\ntgt = pr.SwathDefinition(ecco_lons_2d, ecco_lats_2d)\n\npyresample.geometry.SwathDefinition\n\n\nGrid ssha or other variable\nGet one timestamp to represent the midpoint of the 10-day cycle.\n\ntime = np.datetime64(ds['time'].mean().data)\n\nprint(time)\n\n2021-07-18T15:11:35.073955170\n\n\nAccess the target variable, ssha in this case. Make a nan mask from the ssha variable.\n\nnans = ~np.isnan(ds.ssha.values)\n\nssha = ds.ssha.values[nans]\n\nssha.shape\n\n(518164,)\n\n\nCreate the source swath definition from the 1d arrays of lons and lats from the S6 level-2 along-track altimetry time series.\n\nlons = ds.longitude.values[nans]\nlats = ds.latitude.values[nans]\n \nprint(lons.shape, lats.shape)\n\n(518164,) (518164,)\n\n\n\nlons = (lons + 180) % 360 - 180\n\nsrc = pr.SwathDefinition(lons, lats)\n\npyresample.geometry.SwathDefinition\nResample ssha data using kd-tree gaussian weighting neighbour approach.\n\nresult, stddev, counts = resample_gauss(\n src,\n ssha,\n tgt,\n radius_of_influence=175000, \n sigmas=25000,\n neighbours=100,\n fill_value=np.nan,\n with_uncert=True,\n)\n\nresult.shape\n\n/srv/conda/envs/notebook/lib/python3.9/site-packages/pyresample/kd_tree.py:384: UserWarning: Possible more than 100 neighbours within 175000 m for some data points\n warnings.warn(('Possible more than %s neighbours '\n\n\n(360, 720)\n\n\npyresample.kd_tree.resample_gauss\n\ndef to_xrda(data):\n return xr.DataArray(data,\n dims=['latitude', 'longitude'], \n coords={'time': time, \n 'latitude': ecco_lats, \n 'longitude': ecco_lons})\n\nApply the land/water mask in the numpy array created from the ECCO layer in the steps above. Then, convert the masked numpy array to an xarray data array object named gridded. Print its header.\n\ngrid = to_xrda(result)\n\ngrid.sel(latitude=slice(-67.0, 67.0)).plot(vmin=-0.4, vmax=0.4, cmap=\"jet\", figsize=(18, 7))\n\n<matplotlib.collections.QuadMesh at 0x7fed72fc43a0>\n\n\n\n\n\n\nstddev = to_xrda(stddev)\n\nstddev.sel(latitude=slice(-67.0, 67.0)).plot(robust=True, cmap=\"jet\", figsize=(18, 7))\n\n<matplotlib.collections.QuadMesh at 0x7fed72892520>\n\n\n\n\n\n\ncounts = to_xrda(counts)\n\ncounts.sel(latitude=slice(-67.0, 67.0)).plot(robust=True, cmap=\"jet\", figsize=(18, 7))\n\n<matplotlib.collections.QuadMesh at 0x7fed732a3af0>\n\n\n\n\n\n\n\nExercise\nCalculate area-weighted mean sea level.\n\n\nReferences\nnumpy (https://numpy.org/doc/stable/reference)\n\nnumpy.ndarray.data\n\nnumpy.where\n\nnumpy.isnan\n\ndatetimes\n\nnumpy.sum\n\nnumpy.nansum\n\nxarray (https://xarray.pydata.org/en/stable)\n\nxarray.DataArray\n\nxarray.DataArray.values\n\nxarray.DataArray.mean" + "objectID": "external/xarray.html#analysis", + "href": "external/xarray.html#analysis", + "title": "Introduction to xarray", + "section": "Analysis", + "text": "Analysis\nAs a simple example, let’s try to calculate a mean field for the whole time range.\n\nds.mean(dim='time')\n\nWe can also calculate a zonal mean (averaging over longitude)\n\nds.mean(dim='lon')\n\nOther aggregation methods include min(), max(), std(), along with others.\n\nds.std(dim='time')\n\nThe data we have are in 6h timesteps. This can be resampled to daily or monthly. If you are familiar with pandas, xarray uses the same methods.\n\nds.resample(time='M').mean()\n\n\nds_mon = ds.resample(time='M').mean()\nds_mon\n\nThis is a really short time series but as an example, let’s calculate a monthly climatology (at least for 2 months). For this we can use groupby()\n\nds_clim = ds_mon.groupby(ds_mon.time.dt.month).mean()" + }, + { + "objectID": "external/xarray.html#plot-results", + "href": "external/xarray.html#plot-results", + "title": "Introduction to xarray", + "section": "Plot results", + "text": "Plot results\nFinally, let’s plot the results! This will plot the lat/lon axes of the original ds DataArray.\n\nds_clim.air.sel(month=10).plot()" }, { "objectID": "external/data_access_direct_S3.html", @@ -259,417 +595,319 @@ "text": "Resources\n\nBuild time series from multiple GeoTIFF files\nHvplot/Holoview Colormap\nhttps://git.earthdata.nasa.gov/projects/LPDUR/repos/lpdaac_cloud_data_access/browse\nhttps://git.earthdata.nasa.gov/projects/LPDUR/repos/hls-tutorial/browse\nDirect S3 Data Access - Rough PODAAC ECCO SSH Example\nDirect access to ECCO data in S3 (from us-west-2)\nDirect S3 Data Access with GDAL Virtual Raster Format (VRT)\nDirect S3 Data Access with rioxarray - Clipping Example" }, { - "objectID": "external/nasa_earthdata_authentication.html", - "href": "external/nasa_earthdata_authentication.html", - "title": "Authentication for NASA Earthdata", + "objectID": "how-tos/use_apis.html", + "href": "how-tos/use_apis.html", + "title": "How do I access data using APIs?", "section": "", - "text": "imported on: 2023-05-30\nThis notebook was originally developed for the 2021 Cloud Hackathon, and has been updated with most current approaches." + "text": "API stands for Application Programming Interface, and is basically “like a digital mediator that enables different software applications to talk to each other and exchange information. It defines the methods, parameters, and data formats you can use to access certain features or retrieve data from an application or service.” (How to Use An API)" }, { - "objectID": "external/nasa_earthdata_authentication.html#summary", - "href": "external/nasa_earthdata_authentication.html#summary", - "title": "Authentication for NASA Earthdata", - "section": "Summary", - "text": "Summary\nThis notebook creates a hidden .netrc file (_netrc for Window OS) with Earthdata login credentials in your home directory. This file is needed to access NASA Earthdata assets from a scripting environment like Python.\n\nEarthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\nAuthentication via netrc File\nYou will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. An example of the required content is below.\nmachine urs.earthdata.nasa.gov\nlogin <USERNAME>\npassword <PASSWORD>\n<USERNAME> and <PASSWORD> would be replaced by your actual Earthdata Login username and password respectively." + "objectID": "how-tos/use_apis.html#using-apis", + "href": "how-tos/use_apis.html#using-apis", + "title": "How do I access data using APIs?", + "section": "", + "text": "API stands for Application Programming Interface, and is basically “like a digital mediator that enables different software applications to talk to each other and exchange information. It defines the methods, parameters, and data formats you can use to access certain features or retrieve data from an application or service.” (How to Use An API)" }, { - "objectID": "external/nasa_earthdata_authentication.html#import-required-packages", - "href": "external/nasa_earthdata_authentication.html#import-required-packages", - "title": "Authentication for NASA Earthdata", - "section": "Import Required Packages", - "text": "Import Required Packages\n\nfrom netrc import netrc\nfrom subprocess import Popen\nfrom platform import system\nfrom getpass import getpass\nimport os\n\nThe code below will:\n\ncheck what operating system (OS) is being used to determine which netrc file to check for/create (.netrc or _netrc)\ncheck if you have an netrc file, and if so, varify if those credentials are for the Earthdata endpoint\ncreate a netrc file if a netrc file is not present.\n\n\nurs = 'urs.earthdata.nasa.gov' # Earthdata URL endpoint for authentication\nprompts = ['Enter NASA Earthdata Login Username: ',\n 'Enter NASA Earthdata Login Password: ']\n\n# Determine the OS (Windows machines usually use an '_netrc' file)\nnetrc_name = \"_netrc\" if system()==\"Windows\" else \".netrc\"\n\n# Determine if netrc file exists, and if so, if it includes NASA Earthdata Login Credentials\ntry:\n netrcDir = os.path.expanduser(f\"~/{netrc_name}\")\n netrc(netrcDir).authenticators(urs)[0]\n\n# Below, create a netrc file and prompt user for NASA Earthdata Login Username and Password\nexcept FileNotFoundError:\n homeDir = os.path.expanduser(\"~\")\n Popen('touch {0}{2} | echo machine {1} >> {0}{2}'.format(homeDir + os.sep, urs, netrc_name), shell=True)\n Popen('echo login {} >> {}{}'.format(getpass(prompt=prompts[0]), homeDir + os.sep, netrc_name), shell=True)\n Popen('echo \\'password {} \\'>> {}{}'.format(getpass(prompt=prompts[1]), homeDir + os.sep, netrc_name), shell=True)\n # Set restrictive permissions\n Popen('chmod 0600 {0}{1}'.format(homeDir + os.sep, netrc_name), shell=True)\n\n # Determine OS and edit netrc file if it exists but is not set up for NASA Earthdata Login\nexcept TypeError:\n homeDir = os.path.expanduser(\"~\")\n Popen('echo machine {1} >> {0}{2}'.format(homeDir + os.sep, urs, netrc_name), shell=True)\n Popen('echo login {} >> {}{}'.format(getpass(prompt=prompts[0]), homeDir + os.sep, netrc_name), shell=True)\n Popen('echo \\'password {} \\'>> {}{}'.format(getpass(prompt=prompts[1]), homeDir + os.sep, netrc_name), shell=True)\n\n\nSee if the file was created\nIf the file was created, we’ll see a .netrc file (_netrc for Window OS) in the list printed below. To view the contents from a Jupyter environment, click File on the top toolbar, select Open from Path…, type .netrc, and click Open. The .netrc file will open within the text editor.\n\n!!! Beware, your password will be visible if the .netrc file is opened in the text editor.\n\n\n!ls -al ~/" + "objectID": "how-tos/use_apis.html#how-do-i-know", + "href": "how-tos/use_apis.html#how-do-i-know", + "title": "How do I access data using APIs?", + "section": "How do I know?", + "text": "How do I know?\nHow do I know what API services are available for a dataset?\nSome but not all datasets have API services that can reproject, resample or reformat files on the server file. These How-To’s below show how to find which services are available:\n\nFinding Services using EarthData Search\nAfter selecting a dataset and the needed files using Earthdata Search (tutorial here), when the download files button is selected, it will navigate users to a new page on the sidebar. If the dataset is customizable through APIs, different options will pop up under “Select a data access method” or “Configure data customization options”\n\n\nFinding Services using CMR\nFollow this link to see how to programmatically discover what services are associated with what NASA Earthdata." }, { - "objectID": "tutorials/Above_Ground_Biomass_Density.html", - "href": "tutorials/Above_Ground_Biomass_Density.html", - "title": "Distribution of carbon stocks in a protected area", - "section": "", - "text": "The Global Ecosystem Dynamics Investigation (GEDI) Level 4A Footprint Level Aboveground Biomass Density (AGBD) dataset provides predictions of the aboveground biomass density (AGBD; in Mg/ha) and estimates of the prediction standard error within each sampled geolocated GEDI footprint. GEDI L4A dataset is available for the period starting 2019-04-17 and covers 52 N to 52 S latitudes. GEDI L4A data files are natively in HDF5 format.\nThis tutorial will demonstrate how to compute the distribution of carbon stocks or AGBD for a protected area using the GEDI L4A dataset." + "objectID": "how-tos/use_apis.html#harmony", + "href": "how-tos/use_apis.html#harmony", + "title": "How do I access data using APIs?", + "section": "Harmony", + "text": "Harmony\nHarmony allows you to seamlessly analyze Earth observation data from different NASA data centers with services that offer data reduction and common access mechanisms across all NASA data centers. There is a python client library called harmony-py that exists as a wrapper for the Harmony services, and this notebook gives a brief introduction to the Harmony API services." }, { - "objectID": "tutorials/Above_Ground_Biomass_Density.html#overview", - "href": "tutorials/Above_Ground_Biomass_Density.html#overview", - "title": "Distribution of carbon stocks in a protected area", - "section": "", - "text": "The Global Ecosystem Dynamics Investigation (GEDI) Level 4A Footprint Level Aboveground Biomass Density (AGBD) dataset provides predictions of the aboveground biomass density (AGBD; in Mg/ha) and estimates of the prediction standard error within each sampled geolocated GEDI footprint. GEDI L4A dataset is available for the period starting 2019-04-17 and covers 52 N to 52 S latitudes. GEDI L4A data files are natively in HDF5 format.\nThis tutorial will demonstrate how to compute the distribution of carbon stocks or AGBD for a protected area using the GEDI L4A dataset." + "objectID": "how-tos/use_apis.html#opendap", + "href": "how-tos/use_apis.html#opendap", + "title": "How do I access data using APIs?", + "section": "OPeNDAP", + "text": "OPeNDAP\nOPeNDAP, or the Open-source Project for a Network Data Access Protocol, is a data server that allows for accessing scientific datasets through the public internet. OPeNDAP links can be acquired through dataset landing pages or the common metadata repository. For more information, see the OPeNDAP Data Access Client Tutorials." }, { - "objectID": "tutorials/Above_Ground_Biomass_Density.html#learning-objectives", - "href": "tutorials/Above_Ground_Biomass_Density.html#learning-objectives", - "title": "Distribution of carbon stocks in a protected area", - "section": "Learning Objectives", - "text": "Learning Objectives\n\nUse the earthaccess Python library to search for GEDI L4A granules for time and area of interest. The earthaccess library simplifies access to NASA’s Common Metadata Repository (CMR) Application Programming Interface (API).\nUse NASA OPeNDAP Hyrax to retrieve the GEDI L4A dataset. Hyrax allows direct access to selected variables for the dataset within the spatial-temporal bounds without having to download the whole data file.\nCompute distributions of AGBD across various plant functional types (PFTs) in a study area." + "objectID": "how-tos/use_apis.html#earthaccess", + "href": "how-tos/use_apis.html#earthaccess", + "title": "How do I access data using APIs?", + "section": "earthaccess", + "text": "earthaccess\nearthaccess is a python library to search, download or stream NASA Earth science data with just a few lines of code that also abstracts a lot of other APIs developed by NASA." }, { - "objectID": "tutorials/Above_Ground_Biomass_Density.html#requirements", - "href": "tutorials/Above_Ground_Biomass_Density.html#requirements", - "title": "Distribution of carbon stocks in a protected area", - "section": "Requirements", - "text": "Requirements\n\n1. Compute environment\nThis notebook can be run in any personal computing environment (e.g., desktop/laptops), on-premise solution (e.g., High-Performance Computing), or on the Cloud (e.g., Amazon Web Service). ### 2. Earthdata Login An Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up. ### 3. Additional Requirements\nThis tutorial requires the following Python modules installed in your system: earthaccess, requests, netCDF4, datetime, dask, pandas, geopandas, contextily, shapely, seaborn. To install the necessary Python modules, you can run:\npip install -r earthaccess requests netCDF4 datetime dask pandas geopandas contextily shapely seaborn" - }, - { - "objectID": "tutorials/Above_Ground_Biomass_Density.html#import-packages", - "href": "tutorials/Above_Ground_Biomass_Density.html#import-packages", - "title": "Distribution of carbon stocks in a protected area", - "section": "Import packages", - "text": "Import packages\n\nimport earthaccess\nimport pandas as pd\nimport geopandas as gpd\nimport dask.dataframe as dd\nimport contextily as ctx\nimport netCDF4 as nc\nfrom os import path\nfrom matplotlib import pyplot as plt\nfrom shapely.ops import orient\nfrom IPython import display\nfrom requests.adapters import HTTPAdapter, Retry\nimport seaborn as sns\nsns.set(style='whitegrid')\n\nAccess to NASA OPeNDAP requires NASA Earthdata authentication. We recommend authenticating your Earthdata Login (EDL) information using the earthaccess python library as follows:\n\nauth = earthaccess.login(strategy=\"netrc\") # works if the EDL login already been persisted to a netrc\nif not auth.authenticated:\n # ask for EDL credentials and persist them in a .netrc file\n auth = earthaccess.login(strategy=\"interactive\", persist=True)\n\nYou're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 10/28/2023\nUsing .netrc file for EDL" - }, - { - "objectID": "tutorials/Above_Ground_Biomass_Density.html#search-gedi-l4a-granules", - "href": "tutorials/Above_Ground_Biomass_Density.html#search-gedi-l4a-granules", - "title": "Distribution of carbon stocks in a protected area", - "section": "Search GEDI L4A granules", - "text": "Search GEDI L4A granules\nFirst, we will search for granules that overlay our area of interest. We will then access the location and science variables for the GEDI shots using OPeNDAP.\nFor this tutorial, we will use the boundary of Annapurna Conservation Area (ACA), the largest protected area of Nepal, to search for the overlapping GEDI L4A granules. The boundary polygon is available as a GeoJSON file at aca.json. Let’s plot the boundary polygon of the ACA over a basemap.\n\n# read and plot ACA GeoJSON file\naca = gpd.read_file(\"https://raw.githubusercontent.com/ornldaac/gedi_tutorials/main/polygons/aca.json\") \nax=aca.to_crs(epsg=3857).plot(figsize=(5, 6), edgecolor=\"red\", alpha=0.3)\nctx.add_basemap(ax)\nax.set_title(\"Annapurna Conservation Area, Nepal\")\nplt.show()\n\n\n\n\nWe will now search for all the GEDI L4A granules overlapping the ACA boundary polygon using the Earthaccess Python library. We will also limit the period of our search to the year 2020 (Jan 1 to Dec 31, 2020). The earthaccess’s search_data allows searching dataset granules using NASA’s CMR API. It requires the dataset digital object identifier (DOI), time, and the polygon coordinates as tuples.\n\n# GEDI L4A DOI \ndoi = '10.3334/ORNLDAAC/2056'\n\n# time-bound for the year 2020\nstart_date = \"2020-01-01\" # specify your start date\nend_date = \"2020-12-31\" # specify your end date\n\n# bounding lon, lat as a list of tuples\naca.geometry = aca.geometry.apply(orient, args=(1,))\n# simplifying the polygon to bypass the coordinates \n# limit of the CMR with a tolerance of .005 degrees\nxy = aca.geometry.simplify(0.005).get_coordinates()\ncoord_list = [(x,y) for x,y in zip(xy.x , xy.y)]\n\ngranule_arr = earthaccess.search_data(\n count=-1, # needed to retrieve all granules\n doi=doi,\n polygon=coord_list,\n temporal=(start_date, end_date)\n)\n\nGranules found: 59\n\n\nNow, the GEDI L4A granule information, including data URLs, is stored in granule_arr, from which we can create OPeNDAP URLs. We will use NetCDF-4 encoding in the HTTP requests as an NC4 file representation. Let’s print the first three granules from the list.\n\n# create opendap urls from the granule links\nopendap_base ='https://opendap.earthdata.nasa.gov'\nopendap_url = f'{opendap_base}/collections/C2237824918-ORNL_CLOUD/granules/GEDI_L4A_AGB_Density_V2_1'\nopendap_arr = [f'{opendap_url}.{path.basename(granule.data_links()[0])}.dap.nc4' for granule in granule_arr]\ntotal_granules = len(opendap_arr)\n# print the first three links\nopendap_arr[:3]\n\n['https://opendap.earthdata.nasa.gov/collections/C2237824918-ORNL_CLOUD/granules/GEDI_L4A_AGB_Density_V2_1.GEDI04_A_2020009160947_O06097_02_T00100_02_002_02_V002.h5.dap.nc4',\n 'https://opendap.earthdata.nasa.gov/collections/C2237824918-ORNL_CLOUD/granules/GEDI_L4A_AGB_Density_V2_1.GEDI04_A_2020013143447_O06158_02_T04675_02_002_02_V002.h5.dap.nc4',\n 'https://opendap.earthdata.nasa.gov/collections/C2237824918-ORNL_CLOUD/granules/GEDI_L4A_AGB_Density_V2_1.GEDI04_A_2020024182123_O06331_03_T00197_02_002_02_V002.h5.dap.nc4']" - }, - { - "objectID": "tutorials/Above_Ground_Biomass_Density.html#access-the-variables-of-interest-for-the-subset-area", - "href": "tutorials/Above_Ground_Biomass_Density.html#access-the-variables-of-interest-for-the-subset-area", - "title": "Distribution of carbon stocks in a protected area", - "section": "Access the variables of interest for the subset area", - "text": "Access the variables of interest for the subset area\nOPeNDAP allows access to the dataset of interest, so you can only download the variables you are interested in and, by doing so, save the download bandwidth. For this tutorial, we are interested in the following selected science variables (agbd, l4_quality_flag, and pft_class) for four full-powered beams of the eight GEDI beams. Refer to this tutorial for more information about GEDI L4A data structure.\n\n# variables of interests\nvariables = ['agbd', 'l4_quality_flag', 'land_cover_data/pft_class']\n\n# beams of interest\nbeams = ['BEAM0101', 'BEAM0110', 'BEAM1000', 'BEAM1011']\n\n# setting header variables\nheaders = ['lat_lowestmode', 'lon_lowestmode', 'elev_lowestmode', 'shot_number']\nheaders.extend(variables)\n\nNow, we will retrieve the dataset from the OPeNDAP Hyrax server. Let’s set up a session and set the number of maximum retries.\n\n# setting maximum retries for the HTTP sessions to get around the 500 error\nsession = earthaccess.get_requests_https_session()\nretries = Retry(total=10, backoff_factor=0.1, status_forcelist=[ 500, 502, 503, 504 ])\nsession.mount('https://', HTTPAdapter(max_retries=retries))\n\nFirst, we will access the geographic variables (lat_lowestmode, lon_lowestmode) of all the GEDI shots within the beam to identify the shots overlapping the area of interest. We will then retrieve the science variables of interest (agbd, agbd_t, etc.) for the area of interest.\nThe following step will take some time to run as there are 59 granules to process. The figure below will update to indicate the download status.\n\n# plotting the area of interest\nax=aca.to_crs(epsg=3857).plot(figsize=(5, 6), alpha=0.3)\nctx.add_basemap(ax)\nax.set_title(\"Starting download...\")\ndisplay.display(plt.gcf())\ndisplay.clear_output(wait=True)\n\nc=0\ndfs=[]\n# loop through all granule URLs\nfor g_name in opendap_arr:\n c += 1\n # loop through all beams\n for beam in beams: \n # 1. Retrieving coordinates for the beam\n r = session.get(f\"{g_name}?dap4.ce=/{beam}/lon_lowestmode;/{beam}/lat_lowestmode\")\n if (r.status_code != 400):\n with nc.Dataset('hyrax', memory=r.content) as ds:\n df = pd.DataFrame({'lat_lowestmode': ds[beam]['lat_lowestmode'][:], \n 'lon_lowestmode': ds[beam]['lon_lowestmode'][:]}) # creating pandas dataframe \n\n # 2. Subsetting by bounds of the area of interest\n # converting to geopandas dataframe\n gdf = gpd.GeoDataFrame(df, geometry=gpd.points_from_xy(df.lon_lowestmode, df.lat_lowestmode)) \n gdf_aca = gdf[gdf['geometry'].within(aca.geometry[0])] \n if not gdf_aca.empty:\n # creating empty dataframe columns for variables\n for v in headers[2:]:\n gdf_aca[v] = pd.Series()\n # 3. Retrieving variables of interest. Only retrieving GEDI shots within the subset area.\n for _, df_gr in gdf_aca.groupby((gdf_aca.index.to_series().diff() > 1).cumsum()):\n i = df_gr.index.min()\n j = df_gr.index.max()\n for v in headers[2:]:\n r = session.get(f\"{g_name}?dap4.ce=/{beam}/{v}%5B{i}:{j}%5D\")\n if (r.status_code != 400):\n with nc.Dataset('hyrax', memory=r.content) as ds:\n gdf_aca.loc[i:j, (v)] = ds[beam][v][:]\n\n # creating a dask dataframe\n dfs.append(dd.from_pandas(pd.DataFrame(gdf_aca.drop(columns='geometry')), npartitions=3))\n\n # plotting the GEDI shots on map\n gdf_aca.crs = \"EPSG:4326\"\n gdf_aca.to_crs(epsg=3857).plot(alpha=0.01, ax=ax, linewidth=0)\n ax.set_title(f\"Downloading {c} of {total_granules}... {g_name.rsplit('/', 1)[-1][:-8]} / {beam}\")\n display.display(plt.gcf())\n display.clear_output(wait=True)\n\n\n\n\nLet’s concatenate the dataframes into a single pandas dataframe and print the first few rows. The concatenated dataframe now contains the subset data for the GEDI L4A shots within the ACA boundary, which can also be saved as a CSV file, as shown below (commented out).\n\n# concating the dataframe \nl4a_df = dd.concat(dfs).compute()\n# uncomment the line below to output the data as a CSV file\n# l4a_df.to_csv('aca_gedi_l4a.csv', index=False, header=False, columns=headers)\n# printing the first few rows\nl4a_df.head()\n\n\n\n\n\n\n\n\nlat_lowestmode\nlon_lowestmode\nelev_lowestmode\nshot_number\nagbd\nl4_quality_flag\nland_cover_data/pft_class\n\n\n\n\n69740\n28.533718\n83.651917\n1334.218384\n60970500200160455\n2.014405\n0\n6\n\n\n69741\n28.534097\n83.652311\n1336.080078\n60970500200160456\n63.936985\n0\n4\n\n\n69742\n28.534474\n83.652703\n1344.832642\n60970500200160457\n18.22143\n0\n4\n\n\n69743\n28.534852\n83.653096\n1349.221069\n60970500200160458\n76.016953\n0\n4\n\n\n69744\n28.535227\n83.653488\n1362.543945\n60970500200160459\n23.200373\n0\n4" + "objectID": "how-tos/use_apis.html#appeears", + "href": "how-tos/use_apis.html#appeears", + "title": "How do I access data using APIs?", + "section": "AppEEARS", + "text": "AppEEARS\nThe Application for Extracting and Exploring Analysis Ready Samples (AρρEEARS) offers users a simple and efficient way to perform data access and transformation processes. By enabling users to subset data spatially, temporally, and by layer, the volume of data downloaded for analysis is greatly reduced. AppEEARS requests can be submitted in website and via API. In a same way, outputs generated can also be accessed from AppEEARS website and through API. View AppEEARS documentation and API documentation for more information. Additionally, you can work with the data in the cloud using provided S3 links. Visit AppEEARS Data Resources Repository for more tutorials on how to work with AppEEARS programaticaly." }, { - "objectID": "tutorials/Above_Ground_Biomass_Density.html#distribution-of-agbd-in-annapurna-conservation-area", - "href": "tutorials/Above_Ground_Biomass_Density.html#distribution-of-agbd-in-annapurna-conservation-area", - "title": "Distribution of carbon stocks in a protected area", - "section": "Distribution of AGBD in Annapurna Conservation Area", - "text": "Distribution of AGBD in Annapurna Conservation Area\nThe pft_class variable provides the plant functional types for the shots derived from the MODIS land cover type product (MCD12Q1). It uses LC_Type5 or Annual PFT classification. We can now plot the distribution of the AGBD by plant functional types (PFTs) for the good-quality shots in the ACA region.\n\n# MCD12Q1 PFT types\npft_legend = {0 : 'Water Bodies', 1: 'Evergreen Needleleaf Trees', 2: 'Evergreen Broadleaf Trees', \n 3: 'Deciduous Needleleaf Trees', 4: 'Deciduous Broadleaf Trees', 5: 'Shrub', 6: 'Grass',\n 7: 'Cereal Croplands', 8: 'Broadleaf Croplands', 9: 'Urban and Built-up Lands', \n 10: 'Permanent Snow and Ice', 11: 'Barren', 255: 'Unclassified'}\n\n# creating a mask with good quality shots and trees/shrubs PFT class\nl4a_df_masked = l4a_df[(l4a_df['l4_quality_flag']==1) & (l4a_df['land_cover_data/pft_class'] <= 5 )]\nplt.figure(figsize=(15,5))\n# drawing distribution plots\nax = l4a_df_masked.groupby('land_cover_data/pft_class')['agbd'].\\\n apply(lambda x: sns.distplot(x, label = pft_legend[x.name], hist=True))\nplt.xlabel('agbd (Mg / ha)')\nplt.title(f'Distribution of GEDI L4A AGBD estimates by PFTs (Plant Functional Types) in ACA in 2020')\nplt.legend()\nplt.show()\n\n\n\n\nLet’s plot how the AGBD is distributed across elevation ranges for different PFTs.\n\n# create 500m elevation bins\nl4a_df_masked['elev_bin']=pd.cut(l4a_df_masked['elev_lowestmode'], bins =range(0, 5000, 500))\n# drawing categorical plot\ng = sns.catplot(x = \"elev_bin\", y = \"agbd\", data = l4a_df_masked, col=\"land_cover_data/pft_class\", kind=\"box\")\ng.set_xticklabels(rotation=90)\ng.set_titles(\"{col_name}\")\nfor ax in g.axes.flat:\n ax.set_title(pft_legend[int(float(ax.get_title()))])\ng.set_axis_labels(\"Elevation (m)\")\nsns.despine(top=False, right=False, left=False, bottom=False, offset=None, trim=False)\nplt.show()\n\n\n\n\nAdditional tutorials on discovering, accessing, and using GEDI Level 3 and Level 4 data products are available at https://github.com/ornldaac/gedi_tutorials." + "objectID": "how-tos/find-data/find-python.html", + "href": "how-tos/find-data/find-python.html", + "title": "How do I find data using Python?", + "section": "", + "text": "Here are our recommended approaches for finding data with Python, from the command line or a notebook.\nIn Python we can use the earthaccess library.\nThis example searches for data from the Land Processes DAAC with a spatial bounding box and temporal range.\n\n## Import packages\nimport earthaccess\nfrom earthaccess import DataGranules, DataCollections\nfrom pprint import pprint \n\n\n## We'll get 4 collections that match with our keyword of interest\ncollections = DataCollections().keyword(\"REFLECTANCE\").cloud_hosted(True).get(4)\n\n\n## Let's print 2 collections\nfor collection in collections[0:2]:\n print(pprint(collection.summary()) , collection.abstract(), \"\\n\")\n\n{'cloud-info': {'Region': 'us-west-2',\n 'S3BucketAndObjectPrefixNames': ['s3://lp-prod-protected/MOD09GA.061',\n 's3://lp-prod-public/MOD09GA.061'],\n 'S3CredentialsAPIDocumentationURL': 'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentialsREADME',\n 'S3CredentialsAPIEndpoint': 'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials'},\n 'concept-id': 'C2202497474-LPCLOUD',\n 'file-type': \"[{'Format': 'HDF-EOS2', 'FormatType': 'Native', 'Media': \"\n \"['HTTPS', 'Earthdata Cloud']}]\",\n 'get-data': ['https://e4ftl01.cr.usgs.gov/MOLT/MOD09GA.061/',\n 'https://search.earthdata.nasa.gov/search/granules?p=C2202497474-LPCLOUD',\n 'https://appeears.earthdatacloud.nasa.gov/',\n 'https://earthexplorer.usgs.gov/'],\n 'short-name': 'MOD09GA',\n 'version': '061'}\nNone The MOD09GA Version 6.1 product provides an estimate of the surface spectral reflectance of Terra Moderate Resolution Imaging Spectroradiometer (MODIS) Bands 1 through 7, corrected for atmospheric conditions such as gasses, aerosols, and Rayleigh scattering. Provided along with the 500 meter (m) surface reflectance, observation, and quality bands are a set of ten 1 kilometer (km) observation bands and geolocation flags. The reflectance layers from the MOD09GA are used as the source data for many of the MODIS land products. \n\nValidation at stage 3 (https://modis-land.gsfc.nasa.gov/MODLAND_val.html) has been achieved for the MODIS Surface Reflectance products. Further details regarding MODIS land product validation for the MOD09 data product is available from the MODIS Land Team Validation site (https://modis-land.gsfc.nasa.gov/ValStatus.php?ProductID=MOD09).\n\n Improvements/Changes from Previous Versions\n\n* The Version 6.1 Level-1B (L1B) products have been improved by undergoing various calibration changes that include: changes to the response-versus-scan angle (RVS) approach that affects reflectance bands for Aqua and Terra MODIS, corrections to adjust for the optical crosstalk in Terra MODIS infrared (IR) bands, and corrections to the Terra MODIS forward look-up table (LUT) update for the period 2012 - 2017.\n* A polarization correction has been applied to the L1B Reflective Solar Bands (RSB).\n\n \n\n{'cloud-info': {'Region': 'us-west-2',\n 'S3BucketAndObjectPrefixNames': ['s3://lp-prod-protected/MOD09GQ.061',\n 's3://lp-prod-public/MOD09GQ.061'],\n 'S3CredentialsAPIDocumentationURL': 'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentialsREADME',\n 'S3CredentialsAPIEndpoint': 'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials'},\n 'concept-id': 'C2343115666-LPCLOUD',\n 'file-type': \"[{'Format': 'HDF-EOS2', 'FormatType': 'Native', 'Media': \"\n \"['HTTPS', 'Earthdata Cloud']}]\",\n 'get-data': ['https://e4ftl01.cr.usgs.gov/MOLT/MOD09GQ.061/',\n 'https://search.earthdata.nasa.gov/search?q=C2343115666-LPCLOUD',\n 'https://earthexplorer.usgs.gov/',\n 'https://appeears.earthdatacloud.nasa.gov/'],\n 'short-name': 'MOD09GQ',\n 'version': '061'}\nNone The MOD09GQ Version 6.1 product provides an estimate of the surface spectral reflectance of Terra Moderate Resolution Imaging Spectroradiometer (MODIS) 250 meter (m) bands 1 and 2, corrected for atmospheric conditions such as gasses, aerosols, and Rayleigh scattering. Along with the 250 m surface reflectance bands are the Quality Assurance (QA) layer and five observation layers. This product is intended to be used in conjunction with the quality and viewing geometry information of the 500 m product (MOD09GA). \n\nValidation at stage 3 (https://modis-land.gsfc.nasa.gov/MODLAND_val.html) has been achieved for the MODIS Surface Reflectance products. Further details regarding MODIS land product validation for the MOD09 data product is available from the MODIS Land Team Validation site (https://modis-land.gsfc.nasa.gov/ValStatus.php?ProductID=MOD09).\n\nImprovements/Changes from Previous Versions\n\n* The Version 6.1 Level-1B (L1B) products have been improved by undergoing various calibration changes that include: changes to the response-versus-scan angle (RVS) approach that affects reflectance bands for Aqua and Terra MODIS, corrections to adjust for the optical crosstalk in Terra MODIS infrared (IR) bands, and corrections to the Terra MODIS forward look-up table (LUT) update for the period 2012 - 2017.\n* A polarization correction has been applied to the L1B Reflective Solar Bands (RSB). \n\n\n\n\n## Search for files from the second dataset result over a small plot in Nebraska, USA for two weeks in September 2022\ngranules = earthaccess.search_data(concept_id = \"C2021957657-LPCLOUD\", \n temporal = (\"2022-09-10\",\"2022-09-24\"),\n bounding_box = (-101.67271,41.04754,-101.65344,41.06213)) \n\nGranules found: 8" }, { - "objectID": "tutorials/index.html", - "href": "tutorials/index.html", - "title": "Tutorials", + "objectID": "how-tos/find-data/find-r.html", + "href": "how-tos/find-data/find-r.html", + "title": "How do I find data using R?", "section": "", - "text": "The following tutorials are meant to present example workflows with NASA Earthdata in the cloud from a variety of use cases. The tutorials exemplify the advantages of working with NASA Earthdata in the cloud and aim to provide a starting point for exploring what your workflows could look like in the cloud.\n\n\nEarthdata Cloud Clinic - Tutorial\nTwo different direct cloud access & subsetting options available in the Earthdata Cloud\n\n\nSea Surface Temperature and Wind Speeds during a Hurricane - Tutorial\nObtaining NASA Earthdata from different disciplines and data centers\n\n\nSea Level Rise - Tutorial\nUsing the earthaccess Python package for quick sea level rise analysis in the cloud\n\n\nDistribution of Carbon Stocks in a Protected Area - Tutorial\nUsing earthaccess Python package and OPeNDAP to access predictions of the aboveground biomass density\n\n\nObserving Seasonality in Agricultural Areas - Tutorial\nWorking with HLS Landsat and Sentinel-2 data products in the cloud\n\n\nICESat-2 and Landsat cloud access and data integration - Tutorial\nAccessing multiple large datasets in the cloud\n\n\nPairing Cloud and non-Cloud Data - Tutorial\nAccessing and harmonizing data located within and outside of the NASA Earthdata Cloud\n\n\nearthaccess Python Library Demo - Tutorial\n\n\nUsing OPeNDAP in the Cloud - Tutorial\n\n\nPygeoweaver Workflow Demo - Tutorial\n\nWe also have links to past workshops. Tutorials in these past workshops are developed to teach open science and Cloud workflows for specific audiences. They are a snapshot in time as workflows with NASA Earthdata Cloud emerge and evolve." + "text": "This is a work in progress, and may be split up into different modules/tutorials as we continue to work on it.\nHere are our recommended approaches for finding data with R, from the command line or a notebook.\n\nUsing the web interface\nSee How do I find data using Earthdata Search? - in that tutorial, we found the dataset ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4), with the shortname ECCO_L4_SSH_05DEG_MONTHLY_V4R4.\n\n\nSearching programmatically\nThe NASA cloud data is searchable programmatically via two methods - NASA’s own search service, and the NASA Spatio-Temporal Asset Catalog (STAC) API. To find data in R, we’ll mainly rely on the rstac package. This enables us interact with the NASA STAC API to search for our data, and at the same time learn about STAC more generally. This will be useful as it is a common standard for distributing spatial data.\nWe will also search for data using the NASA Earthdata search API, which is the service that powers the Earthdata web search portal.\nFor both of these services, the earthdatalogin package will help us get set up and provide a few handy functions to smooth some rough edges.\n\n\nAuthentication for NASA Earthdata\nAn Earthdata Login account is required to access data from the NASA Earthdata system. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\nOnce you have created your Earthdata Login account, you can use the earthdatalogin R package to help you manage your authentication within R.\nThere is some functionality currently only available in the development version of earthdatalogin, so we will install it from GitHub using the pak package:\ninstall.packages(\"pak\")\npak::pak(\"boettiger-lab/earthdatalogin\")\nThe easiest and most portable method of access is using the netrc basic authentication protocol for HTTP. Call edl_netrc() to set this up given your username and password. The easiest way to store your credentials for use in edl_netrc() is to set them as environment variables in your ~/.Renviron file. The usethis package has a handy function to find and open this file for editing:\n# install.packages(\"usethis\")\nusethis::edit_r_environ()\nThen add the following lines to the file, save the file, and restart R.\nEARTHDATA_USER=\"your_user_name\"\nEARTHDATA_PASSWORD=\"your_password\"\nNow, when you call edl_netrc(), it will consult these environment variables to create a .netrc file that will be used to authenticate with the NASA Earthdata services. If you don’t have your credentials saved as environment variables, earthdatalogin will provide its own credentials, but you may experience rate limits more readily. You can also manually type in your credentials to the username and password arguments to edl_netrc() but this is not recommended as it is too easy to accidentally share these in your code.\n\nlibrary(earthdatalogin)\n\nedl_netrc()\n\nOnce edl_netrc() has been called in your R session, then most spatial packages in R can seamlessly access NASA Earthdata over HTTP links.\n\n\nFinding data in NASA STACs with rstac\nAll of the NASA STAC catalogues can be viewed here: https://radiantearth.github.io/stac-browser/#/external/cmr.earthdata.nasa.gov/stac/.\nWe will use the rstac package to first browse the collections in the PO DAAC catalogue (POCLOUD):\n\n## In R\n## load R libraries\n# install.packages(\"pak\") \n# pak::pak(c(\"tidyverse\", \"rstac\", \"boettiger-lab/earthdatalogin\"))\n\nlibrary(rstac)\nlibrary(earthdatalogin)\n\npo_collections <- stac(\"https://cmr.earthdata.nasa.gov/stac/POCLOUD/\") |>\n collections() |>\n get_request()\n\npo_collections\n\nThis only gives us the first 10 collections in the catalogue; to get the rest we can use the collections_fetch() function:\n\nall_po_collections <- collections_fetch(po_collections)\nall_po_collections\n\nlength(all_po_collections$collections)\n\nhead(all_po_collections$collections)\n\n# Just look at the titles:\nsapply(all_po_collections$collections, `[[`, \"title\")\n\n# Get shortnames from the 'id' field and search for a match:\ngrep(\"ECCO_L4_SSH\", sapply(all_po_collections$collections, `[[`, \"id\"), value = TRUE)\n\nOnce we have searched through the collections, we can choose the one we are interested in and query it to get the items (granules) we are interested in:\n\nstart <- \"2015-01-01\"\nend <- \"2015-12-31\" \n\nitems <- stac(\"https://cmr.earthdata.nasa.gov/stac/POCLOUD\") |> \n stac_search(collections = \"ECCO_L4_SSH_05DEG_MONTHLY_V4R4\",\n datetime = paste(start,end, sep = \"/\")) |>\n post_request() |>\n items_fetch()\n\nitems\n\nThere are 13 ‘items’ representing the same 13 granules we found when we searched using EarthData Search.\nWe can see more details of the items (granules) by printing the features list item:\n\nitems$features\n\n# And the urls:\nedl_stac_urls(items)\n\n\n\nFinding data in NASA EarthData Search using earthdatalogin\nOnce we know the shortname of a collection (usually by looking in the EarthData Search portal), we can supply it to edl_search() to get the metadata and file urls of the individual granules:\n\ngranules <- edl_search(\n short_name = \"ECCO_L4_SSH_05DEG_MONTHLY_V4R4\",\n temporal = c(start, end), \n parse_results = FALSE\n)\n\ngranules\n\ngranules[[1]]\n\n# See the granule titles\nsapply(granules, `[`, \"title\")\n\n# Note these are the same urls obtained via the rstac workflow demonstrated above\ngranule_urls <- edl_extract_urls(granules)\ngranule_urls\n\n\nAccessing data using the {terra} package\nWe can read any of these urls using terra::rast(). We supply the vsi = TRUE argument, which prepends \"/vsicurl/\" to the url, indicating that the file should be opened as a “virtual” remote dataset. This allows random partial reading of files without prior download of the entire file. This will vastly speed up most operations as only the subset of the data required is ever actually downloaded.\n\nlibrary(terra)\n\nrast <- terra::rast(granule_urls[1], vsi = TRUE)\n\n# This does not come with an extent and CRS embedded so we can supply it manually\ngranules[[1]]$boxes\next(rast) <- c(-180, 180, -90, 90)\ncrs(rast) <- \"EPSG:4326\"\n\nplot(rast[[\"SSH\"]])\n\n# We notice that this plot is upside-down - it is likely that the NetCDF file\n# does not conform properly to the conventions. But we can flip it:\nrast <- flip(rast, direction = \"vertical\")\nplot(rast)\n\nIf we want to crop the raster, we can define an extent and crop it to that area. Because we previously called edl_netrc(), we are not only authenticated with the server so we can access the data, but we have also set up terra and stars to us the underlying GDAL library to access the data in such a way that only subset of the data we have requested is actually downloaded.\n\ncrop_box <- rast(extent = c(-150, -120, 35, 60), crs = \"EPSG:4326\")\n\nrast_cropped <- crop(rast, crop_box)\n\nplot(rast_cropped[[\"SSH\"]])\n\n\n\nAccessing data using the {stars} package\nThe read_* functions in stars do not have the vsi argument, but we can do the same thing simply by prepending \"/vsicurl/\" to the url ourselves. Here we will use the read_mdim() function.\n\nlibrary(stars)\nssh_stars <- read_mdim(paste0(\"/vsicurl/\", granule_urls[1]))\n\nplot(ssh_stars)\n\nWe can again crop this using the same bounding box as we did with terra:\n\nst_crop(\n ssh_stars, \n st_bbox(c(xmin = -150, xmax = -120, ymin = 35, ymax = 60))\n) |> \n plot()\n\n\n\n\nAccessing data using gdalcubes\nComing soon!" }, { - "objectID": "tutorials/Earthdata-cloud-clinic.html", - "href": "tutorials/Earthdata-cloud-clinic.html", - "title": "NASA Earthdata Cloud Clinic", + "objectID": "how-tos/reproject_regrid.html", + "href": "how-tos/reproject_regrid.html", + "title": "How do I reproject and resample a data file?", "section": "", - "text": "Welcome to the NASA Earthdata Cloud Clinic!\nWe will focus on NASA Earthdata search & access in the cloud.\n\nWe will use earthaccess for data search and direct cloud access, followed by xarray for subsetting. Both are open source python libraries. We will also discover data using Earthdata Search.\nWe will be accessing data directly from the NASA Earthdata Cloud hosted in Amazon Web Services (AWS) Cloud, specifically in the us-west-2 region. This shared compute environment (JupyterHub) is also running in the same location in the AWS Cloud. We will then load the data into Python as an xarray dataset.\nSee second part of this notebook for examples using subsetting servises to downselect and work with data for your use case.\nSee the bottom of the notebook for additional resources, including several tutorials that served as a foundation for this clinic.\n\n\n\nIn this example we will use the earthaccess library to search for data collections from NASA Earthdata. earthaccess is a Python library that simplifies data discovery and access to NASA Earth science data by providing an abstraction layer for NASA’s Common Metadata Repository (CMR) API Search API. The library makes searching for data more approachable by using a simpler notation instead of low level HTTP queries. earthaccess takes the trouble out of Earthdata Login authentication, makes search easier, and provides a stream-line way to download or stream search results into an xarray object for easy data access. It can be used on and off the cloud.\nFor more on earthaccess visit the earthaccess documentation site. Be aware that earthaccess is under active development, and your use and feedback help improve it!\n\n\n\nIn addition to directly accessing the files archived and distributed by each of the NASA DAACs, many datasets also support services that allow us to customize the data via subsetting, reformatting, reprojection/regridding, and file aggregation. What does subsetting mean? Here’s a generalized graphic of what we mean.\n\n\n\n\n“direct cloud access” goes by several other names including “direct S3 access”, “direct access”, “direct in-region access”, “in-cloud data streaming”. And “subsetting” is also called “transformation”." + "text": "Coming Soon: brief introduction. Define reprojecting & resampling" }, { - "objectID": "tutorials/Earthdata-cloud-clinic.html#summary", - "href": "tutorials/Earthdata-cloud-clinic.html#summary", - "title": "NASA Earthdata Cloud Clinic", - "section": "", - "text": "Welcome to the NASA Earthdata Cloud Clinic!\nWe will focus on NASA Earthdata search & access in the cloud.\n\nWe will use earthaccess for data search and direct cloud access, followed by xarray for subsetting. Both are open source python libraries. We will also discover data using Earthdata Search.\nWe will be accessing data directly from the NASA Earthdata Cloud hosted in Amazon Web Services (AWS) Cloud, specifically in the us-west-2 region. This shared compute environment (JupyterHub) is also running in the same location in the AWS Cloud. We will then load the data into Python as an xarray dataset.\nSee second part of this notebook for examples using subsetting servises to downselect and work with data for your use case.\nSee the bottom of the notebook for additional resources, including several tutorials that served as a foundation for this clinic.\n\n\n\nIn this example we will use the earthaccess library to search for data collections from NASA Earthdata. earthaccess is a Python library that simplifies data discovery and access to NASA Earth science data by providing an abstraction layer for NASA’s Common Metadata Repository (CMR) API Search API. The library makes searching for data more approachable by using a simpler notation instead of low level HTTP queries. earthaccess takes the trouble out of Earthdata Login authentication, makes search easier, and provides a stream-line way to download or stream search results into an xarray object for easy data access. It can be used on and off the cloud.\nFor more on earthaccess visit the earthaccess documentation site. Be aware that earthaccess is under active development, and your use and feedback help improve it!\n\n\n\nIn addition to directly accessing the files archived and distributed by each of the NASA DAACs, many datasets also support services that allow us to customize the data via subsetting, reformatting, reprojection/regridding, and file aggregation. What does subsetting mean? Here’s a generalized graphic of what we mean.\n\n\n\n\n“direct cloud access” goes by several other names including “direct S3 access”, “direct access”, “direct in-region access”, “in-cloud data streaming”. And “subsetting” is also called “transformation”." + "objectID": "how-tos/reproject_regrid.html#reproject-with-harmony", + "href": "how-tos/reproject_regrid.html#reproject-with-harmony", + "title": "How do I reproject and resample a data file?", + "section": "Reproject with Harmony", + "text": "Reproject with Harmony" }, { - "objectID": "tutorials/Earthdata-cloud-clinic.html#learning-objectives", - "href": "tutorials/Earthdata-cloud-clinic.html#learning-objectives", - "title": "NASA Earthdata Cloud Clinic", - "section": "Learning Objectives", - "text": "Learning Objectives\n\nUtilize the earthaccess python library to search for data using spatial and temporal filters and explore search results\nStream data (i.e. perform in-region direct access of data) from an Amazon Simple Storage Service (S3) bucket where NASA Earthdata data is archived into our own cloud workspace, here in the Jupyter Lab/Notebook.\nExtract variables and spatial slices from an xarray dataset\nPlot data\nConceptualize data subsetting services provided by NASA Earthdata, including Harmony\nPlot a polygon geojson file with a basemap using geoviews" + "objectID": "how-tos/reproject_regrid.html#reproject-netcdf4", + "href": "how-tos/reproject_regrid.html#reproject-netcdf4", + "title": "How do I reproject and resample a data file?", + "section": "Reproject netcdf4", + "text": "Reproject netcdf4" }, { - "objectID": "tutorials/Earthdata-cloud-clinic.html#prerequisites", - "href": "tutorials/Earthdata-cloud-clinic.html#prerequisites", - "title": "NASA Earthdata Cloud Clinic", - "section": "Prerequisites", - "text": "Prerequisites\nFirst we’ll import python packages and set our authentication that will be used for both of our access and subsetting methods.\nThis tutorial is meant to be run in the AWS cloud in the us-west-2 region. You’ll need to be aware that data in NASA’s Earthdata Cloud reside in Amazon Web Services (AWS) Simple Storage Service (S3) buckets. Access is provided via temporary credentials; this free access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region. While this compute location is required for direct S3 access, all data in Earthdata Cloud are still freely available via download.\n\nImport Required Packages\n\n# Suppress warnings\nimport warnings\nwarnings.simplefilter('ignore')\nwarnings.filterwarnings('ignore')\n\n# Search & Access\nimport earthaccess \nfrom pprint import pprint\nimport xarray as xr\nimport hvplot.xarray #plot\n\n# Harmony services\nimport geopandas as gpd\nimport geoviews as gv\ngv.extension('bokeh', 'matplotlib')\nfrom harmony import BBox, Client, Collection, Request, LinkType\nimport datetime as dt\nimport s3fs\n%matplotlib inline\n\n\n\nAuthentication for NASA Earthdata\nAn Earthdata Login account is required to access data from the NASA Earthdata system. If you don’t already have one, visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\nThe first step is to get the correct authentication that will allow us to get cloud-hosted data from NASA. This is all done through Earthdata Login. We can use the earthaccess library here, where the login method also gets the correct AWS credentials.\nThe first time you run this it will ask you for your Earthdata Login username and password, and stores it in a .netrc file. After that, each time you authenticate with auth = earthaccess.login() it will log you in automatically.\n\n# auth = earthaccess.login(strategy=\"interactive\", persist=True)\nauth = earthaccess.login()" + "objectID": "how-tos/reproject_regrid.html#reproject-geotiff", + "href": "how-tos/reproject_regrid.html#reproject-geotiff", + "title": "How do I reproject and resample a data file?", + "section": "Reproject GeoTIFF", + "text": "Reproject GeoTIFF" }, { - "objectID": "tutorials/Earthdata-cloud-clinic.html#earthaccess-xarray", - "href": "tutorials/Earthdata-cloud-clinic.html#earthaccess-xarray", - "title": "NASA Earthdata Cloud Clinic", - "section": "1. earthaccess + xarray", - "text": "1. earthaccess + xarray\nearthaccess python library is an open-source library to simplify Earthdata Cloud search and access.\n\nSearch for data\nThere are multiple keywords we can use to discovery data from collections such as short_name, concept_id, and doi. The table below contains the short_name for some collections we are interested in for other exercises. Each of these can be used to search for data or information related to the collection we are interested in.\n\n\n\n\n\n\n\n\nShortname\nDescription\nExample Temporal/Spatial parameters\n\n\n\n\nSEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205\ngridded Sea Surface Height Anomalies (SSHA) above a mean sea surface, on a 1/6th degree grid every 5 days.\ntemporal=(“2021-07-01”, “2021-09-30”)\n\n\nMUR25-JPL-L4-GLOB-v04.2\nMUR Sea Surface Temperature\ntemporal=(“2023-07-01”, “2023-07-03”)\n\n\nSMAP_RSS_L3_SSS_SMI_8DAY-RUNNINGMEAN_V5\nSMAP Sea Surface Salinity 8 day running mean\ntemporal=(“2019-08-01T00:00:00”, “2019-10-01T00:00:00”)\n\n\n\nBut wait…You may be asking “how can we find the short_name for collections not in the table above?”.\n–> Let’s take a quick detour and head to Earthdata Search GUI to gather more information about our dataset of interest. The dataset “short name” can be found by clicking on the Info button on our collection search result, and we can paste that into a python variable.\n(Side Note: Both earthaccess python libray and the Earthdata Search (GUI) leverage the Common Metadata Repository (CMR) API to search for collections and granules.)\nHere we use the search_data function of earthaccess to query based on the short_name of interest, as well as other paramters such as temporal range:\n\ndata_name = \"SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205\"\n\nresults = earthaccess.search_data(\n short_name=data_name,\n cloud_hosted=True,\n temporal=(\"2021-07-01\", \"2021-09-30\"),\n)\n\nAccording to PO.DAAC’s dataset landing page, gridded Sea Surface Height Anomalies (SSHA) above a mean sea surface are provided. The data are produced on a 1/6th degree grid every 5 days.\nWe can discover more information about the matching files:\n\npprint(results[0])\n\n\n\nAccess data\nSince we are working in the AWS us-west-2 region, we can stream data directly to xarray. We are using the open_mfdataset() (multi-file) method, which is required when using earthaccess.\n(Tips: To open a single file, if troubleshooting for example: ds = xr.open_dataset(earthaccess.open(results)[0]))\n\nds = xr.open_mfdataset(earthaccess.open(results))\nds\n\n\nPlot the data\nLet’s make a quick interactive plot of the data using an open source tool call hvplot. Because our data is 3d and has a time component, we can also preview the data over time, using the slider of hvplot.\n\nds.SLA.hvplot.image(x='Longitude', y='Latitude', cmap='RdBu', clim=(-2,2), title=\"Sea Level Anomaly Estimate (m)\")\n\n\n\n\nSubset data\nWe can create a subsetted xarray dataset by extracting the SLA variable and slicing the dataset by a smaller area of interest near the state of Baja, Mexico.\n\nds_subset = ds['SLA'].sel(Latitude=slice(15.8, 35.9), Longitude=slice(234.5,260.5)) \nds_subset\n\n\nds_subset.hvplot.image(x='Longitude', y='Latitude', cmap='RdBu', clim=(-0.5, 0.5), title=\"Sea Level Anomaly Estimate (m)\")" + "objectID": "how-tos/reproject_regrid.html#resample-with-hamony", + "href": "how-tos/reproject_regrid.html#resample-with-hamony", + "title": "How do I reproject and resample a data file?", + "section": "Resample with Hamony", + "text": "Resample with Hamony" }, { - "objectID": "tutorials/Earthdata-cloud-clinic.html#exercise", - "href": "tutorials/Earthdata-cloud-clinic.html#exercise", - "title": "NASA Earthdata Cloud Clinic", - "section": "Exercise", - "text": "Exercise\nLet’s explore other datasets from the table above\n\nMUR SST\n\ndata_name = \"MUR25-JPL-L4-GLOB-v04.2\"\n\nsst_results = earthaccess.search_data(\n short_name=data_name,\n cloud_hosted=True,\n temporal=(\"2023-07-01\", \"2023-07-03\"), \n)\n\n\npprint(sst_results[0])\n\n\nds = xr.open_mfdataset(earthaccess.open(sst_results))\nds\n\n\nds.sst_anomaly.hvplot.image(x='lon', y='lat', cmap='RdBu', clim=(-5,5), title=\"Sea Surface Temperature Anomaly (K)\")\n\n\n\nSMAP Sea Surface Salinity\n\ndata_name=\"SMAP_RSS_L3_SSS_SMI_8DAY-RUNNINGMEAN_V5\"\n\nsmap_results = earthaccess.search_data(\n short_name=data_name,\n cloud_hosted=True,\n temporal=(\"2019-08-01T00:00:00\", \"2019-10-01T00:00:00\"),\n)\n\n\npprint(smap_results[0])\n\n\nds = xr.open_mfdataset(earthaccess.open(smap_results))\nds\n\n\nds.sss_smap.hvplot.image(x='lon', y='lat', cmap='plasma', clim=(20, 32), title=\"SMAP_RSS_L3_SSS_SMI_8DAY-RUNNINGMEAN_V5\")\n\n\nds_subset = ds['sss_smap'].sel(lat=slice(61, 75), lon=slice(160,170)) \nds_subset\n\n\nds_subset.hvplot.image(x='lon', y='lat', cmap='plasma', clim=(0, 45), title=\"SMAP_RSS_L3_SSS_SMI_8DAY-RUNNINGMEAN_V5\")\n\n\n\n\nSpatial search tip:\nTo narrow down your search based on a geographic bounding box, we can pass the bounding_box= paramters to earthaccess, as in the example below: #bounding_box=(lower_left_lon, lower_left_lat, upper_right_lon, upper_right_lat) bounding_box=(-99, 18.19232, -78.85547, 31.23754)" + "objectID": "how-tos/reproject_regrid.html#resample-netcdf4", + "href": "how-tos/reproject_regrid.html#resample-netcdf4", + "title": "How do I reproject and resample a data file?", + "section": "Resample netcdf4", + "text": "Resample netcdf4" }, { - "objectID": "tutorials/Earthdata-cloud-clinic.html#harmony-py-xarray", - "href": "tutorials/Earthdata-cloud-clinic.html#harmony-py-xarray", - "title": "NASA Earthdata Cloud Clinic", - "section": "2. Harmony-py + xarray", - "text": "2. Harmony-py + xarray\nBonus cloud clinic topic that we won’t cover live together: subsetting with Earthdata Harmony API services\nWe can also use the Harmony-py python library for direct cloud access & data subsetting: * Requesting a subset of data from the GHRSST Level 4 MUR Global Foundation Sea Surface Temperature Analysis (v4.1) dataset using a vector-based geospatial file. * This dataset can also be viewed in Earthdata Search.\nharmony-py + xarray\nWe will be accessing a subsetted NetCDF-4 file using Transformation Services in the Cloud.\nUsing Earthdata Search, we can find datasets that support these services using the “Customizable” filter.\nWe will find, request, and open customized data using Harmony, below.\n\nDefine area of interest\nFirst, use geopandas to read in a geojson file containing a polygon feature over the Gulf of Mexico. The geojson file is found in the ~/data directory.\n\ngeojson_path = './data/gulf.json'\ngdf = gpd.read_file(geojson_path) #Return a GeoDataFrame object\n\nWe can plot the polygon using the geoviews package that we imported as gv with ‘bokeh’ and ‘matplotlib’ extensions. The following has reasonable width, height, color, and line widths to view our polygon when it is overlayed on a base tile map. We can view this with the * operator.\n\nbase = gv.tile_sources.EsriImagery.opts(width=650, height=500)\nocean_map = gv.Polygons(gdf).opts(line_color='yellow', line_width=5, color=None)\nbase * ocean_map\n\n\n\nCreate a subset request\nHere we’ll create a request for a spatial subset of data.\nFirst, we need to create a Harmony Client, which is what we will interact with to submit and inspect a data request to Harmony, as well as to retrieve results.\nWhen creating the Client, we need to provide Earthdata Login credentials. This basic line below assumes that we have a .netrc available. See the Earthdata Cloud Cookbook appendix for more information on Earthdata Login and netrc setup.\n\nharmony_client = Client()\n\nSee the harmony-py documentation for details on how to construct your request.\n\nsst_short_name=\"MUR-JPL-L4-GLOB-v4.1\"\n\nrequest = Request(\n collection=Collection(id=sst_short_name),\n shape=geojson_path,\n temporal={\n 'start': dt.datetime(2021, 8, 1, 1),\n 'stop': dt.datetime(2021, 8, 1, 2) \n },\n)\n\n\n\nSubmit a subset request\nNow that the request is created, we can now submit it to Harmony using the Harmony Client object. A job id is returned, which is a unique identifier that represents the submitted request.\n\njob_id = harmony_client.submit(request)\njob_id\n\n\n\nCheck request status\nDepending on the size of the request, it may be helpful to wait until the request has completed processing before the remainder of the code is executed. The wait_for_processing() method will block subsequent lines of code while optionally showing a progress bar.\n\nharmony_client.wait_for_processing(job_id, show_progress=True)\n\nWe can view Harmony job response and output URLs Once the data request has finished processing, we can view details on the job that was submitted to Harmony, including the API call to Harmony, and informational messages on the request if available.\nresult_json() calls wait_for_processing() and returns the complete job in JSON format once processing is complete.\n\ndata = harmony_client.result_json(job_id)\npprint(data)\n\n\n\nAccess data\nJust like above, the subsetted outputs produced by Harmony can be accessed directly from the cloud.\nFirst, we’ll retrieve list of output URLs.\nThe result_urls() method calls wait_for_processing() and returns a list of the processed data URLs once processing is complete. You may optionally show the progress bar as shown below.\n\nresults = harmony_client.result_urls(job_id, link_type=LinkType.s3)\nurls = list(results)\nurl = urls[0]\nprint(url)\n\nUsing aws_credentials you can retrieve the credentials needed to access the Harmony s3 staging bucket and its contents.\n\ncreds = harmony_client.aws_credentials()\n\n\n\nOpen staged files with s3fs and xarray\nWe use the AWS s3fs package to create a file system that can then be read by xarray:\n\ns3_fs = s3fs.S3FileSystem(\n key=creds['aws_access_key_id'],\n secret=creds['aws_secret_access_key'],\n token=creds['aws_session_token'],\n client_kwargs={'region_name':'us-west-2'},\n)\n\nNow that we have our s3 file system set, including our declared credentials, we’ll use that to open the url, and read in the file through xarray. This extra step is needed because xarray cannot open the S3 location directly. Instead, the S3 file object is passed to xarray, in order to then open the dataset.\n\nf = s3_fs.open(url, mode='rb')\nds = xr.open_dataset(f)\nds\n\nAs before, we use the xarray built in plotting function to create a simple plot along the x and y dimensions of the dataset. We can see that the data are subsetted to our polygon:\n\nds.analysed_sst.plot();" + "objectID": "how-tos/reproject_regrid.html#resample-geotiff", + "href": "how-tos/reproject_regrid.html#resample-geotiff", + "title": "How do I reproject and resample a data file?", + "section": "Resample GeoTIFF", + "text": "Resample GeoTIFF" }, { - "objectID": "tutorials/Earthdata-cloud-clinic.html#additional-resources", - "href": "tutorials/Earthdata-cloud-clinic.html#additional-resources", - "title": "NASA Earthdata Cloud Clinic", - "section": "Additional Resources", - "text": "Additional Resources\n\nTutorials\nThis clinic was based off of several notebook tutorials including those presented during past workshop events, along with other materials co-created by the NASA Openscapes mentors: * 2021 Earthdata Cloud Hackathon * 2021 AGU Workshop * Accessing and working with ICESat-2 data in the cloud * Analyzing Sea Level Rise Using Earth Data in the Cloud\n\n\nCloud services\nThe examples used in the clinic provide an abbreviated and simplified workflow to explore access and subsetting options available through the Earthdata Cloud. There are several other options that can be used to interact with data in the Earthdata Cloud including:\n\nOPeNDAP\n\nHyrax provides direct access to subsetting of NASA data using Python or your favorite analysis tool\nTutorial highlighting OPeNDAP usage: https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/working-locally/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html\n\nZarr-EOSDIS-Store\n\nThe zarr-eosdis-store library allows NASA EOSDIS Collections to be accessed efficiently by the Zarr Python library, provided they have a sidecar DMR++ metadata file generated.\nTutorial highlighting this library’s usage: https://nasa-openscapes.github.io/2021-Cloud-Hackathon/tutorials/09_Zarr_Access.html\n\n\n\n\nSupport\n\nEarthdata Forum\n\nUser Services and community support for all things NASA Earthdata, including Earthdata Cloud\n\nEarthdata Webinar series\n\nWebinars from DAACs and other groups across EOSDIS including guidance on working with Earthdata Cloud\nSee the Earthdata YouTube channel for more videos" + "objectID": "how-tos/access-data/access-local-opendap.html", + "href": "how-tos/access-data/access-local-opendap.html", + "title": "How do I access Cloud Data from my Local Machine with OPeNDAP?", + "section": "", + "text": "OPeNDAP, or the Open-source Project for a Network Data Access Protocol, is a data server that allows for accessing scientific datasets through the public internet. OPeNDAP links can be acquired through dataset landing pages or the common metadata repository. Once you have the desired link(s), you can open them as follows:\n#Import packages\nimport xarray as xr\n\nds_https = xr.open_dataset(https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_hi_tmax_2010.nc)\n\n# For datasets that contain dap4 specific datatypes, such as `int64`, replace `https` with `dap4` at the beginning to open with xarray. For example:\nurl = https://opendap.earthdata.nasa.gov/collections/C2036881966-POCLOUD/granules/AQUA_MODIS.20220109_20220116.L3m.8D.SST4.sst4.4km\ndap4_url = url.replace(\"https://\", \"dap4://\")\n# dap4_url output: dap4://opendap.earthdata.nasa.gov/collections/C2036881966-POCLOUD/granules/AQUA_MODIS.20220109_20220116.L3m.8D.SST4.sst4.4km\nds_dap4 = xr.open_dataset(dap4_url)\n\n# Note: Some datasets have their variables organized in groups, so within the open_dataset function, you may need to specify 'group=' to access the dataset.\nFor more information, see the OPeNDAP Data Access Client Tutorials." }, { - "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html", - "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html", - "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", + "objectID": "how-tos/access-data/access-local-opendap.html#opendap", + "href": "how-tos/access-data/access-local-opendap.html#opendap", + "title": "How do I access Cloud Data from my Local Machine with OPeNDAP?", "section": "", - "text": "This notebook demonstrates how to access OPeNDAP granules hosted inside of the Earthdata Cloud Archives. It shows how to query a cloud OPeNDAP-hosted Daymet granule using the Common Metadata Repository (CMR) API, before remotely accessing and analyzing it using Xarray and ncdump." + "text": "OPeNDAP, or the Open-source Project for a Network Data Access Protocol, is a data server that allows for accessing scientific datasets through the public internet. OPeNDAP links can be acquired through dataset landing pages or the common metadata repository. Once you have the desired link(s), you can open them as follows:\n#Import packages\nimport xarray as xr\n\nds_https = xr.open_dataset(https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_hi_tmax_2010.nc)\n\n# For datasets that contain dap4 specific datatypes, such as `int64`, replace `https` with `dap4` at the beginning to open with xarray. For example:\nurl = https://opendap.earthdata.nasa.gov/collections/C2036881966-POCLOUD/granules/AQUA_MODIS.20220109_20220116.L3m.8D.SST4.sst4.4km\ndap4_url = url.replace(\"https://\", \"dap4://\")\n# dap4_url output: dap4://opendap.earthdata.nasa.gov/collections/C2036881966-POCLOUD/granules/AQUA_MODIS.20220109_20220116.L3m.8D.SST4.sst4.4km\nds_dap4 = xr.open_dataset(dap4_url)\n\n# Note: Some datasets have their variables organized in groups, so within the open_dataset function, you may need to specify 'group=' to access the dataset.\nFor more information, see the OPeNDAP Data Access Client Tutorials." }, { - "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#import-packages", - "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#import-packages", - "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", - "section": "1. Import Packages", - "text": "1. Import Packages\n\nimport xarray as xr\nimport requests\nimport earthaccess\nimport datetime as dt\nimport pprint\nimport netCDF4 as nc4\nfrom subprocess import Popen\nimport platform\nimport os\nimport shutil" + "objectID": "how-tos/how_to_guide_parking_lot.html", + "href": "how-tos/how_to_guide_parking_lot.html", + "title": "How-To Guides Parking Lot", + "section": "", + "text": "How-To Guides Parking Lot\nThis is a document to capture and organize How-To Guides that do not, yet, have a place in the Cookbook. I’ve pulled examples from the Hackday Google Doc\nNot all of these how-tos may be required. Some may be redundant. Some may be too complicated and could be broken into several How-Tos. Feel free to add comments, links, notes, etc under each suggested How-To. Also suggest changes to the wording. Following the Divio documentation guide, I’ve tried to name how-tos with descriptive names that start “How do I… ?”.\nHow do I reformat data files?\nHow do I reproject and regrid?\nHow do I aggregate data?\nHow do I plot data?\nHow do I use Harmony to reproject a data file?\nHow do I search the USGS Landsat Cloud?\nHow do I access data in the USGS Landsat Cloud?\nHow do I find OPeNDAP datasets?\nHow do I open an OPeNDAP file in the cloud? Include Authentication?\nHow do I open/access an OPeNDAP file from my local machine?\nHow do I open a Zarr file in the cloud?\nHow do I open a COG (Cloud-Optimized-GeoTIFF) in the cloud?\nHow do I use Kerchunk to reformat files? needs refining\nHow do I use CMR STAC to access data in the cloud?\nHow do I use Matlab in the cloud?\nHow do I add a python package to a 2i2c environment?\nHow do I reproduce a shared cloud environment on my local machines" }, { - "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#create-edl-files-using-the-earthaccess-python-library", - "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#create-edl-files-using-the-earthaccess-python-library", - "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", - "section": "2. Create EDL files using the earthaccess Python library", - "text": "2. Create EDL files using the earthaccess Python library\nFirst, pass your Earthdata credentials to the earthaccess library to create the .netrc file:\n\nauth = earthaccess.login(strategy=\"interactive\", persist=True) \n\nYou're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 07/14/2023\nUsing user provided credentials for EDL\nPersisting credentials to .netrc\n\n\nNext, run the following code to generate the .dodsrc file, if it is not already present:\n\nhomeDir = os.path.expanduser(\"~\") + os.sep\n\nwith open(homeDir + '.dodsrc', 'w') as file:\n file.write('HTTP.COOKIEJAR={}.urs_cookies\\n'.format(homeDir))\n file.write('HTTP.NETRC={}.netrc'.format(homeDir))\n file.close()\n\nprint('Saved .dodsrc to:', homeDir)\n\n# Set appropriate permissions for Linux/macOS\nif platform.system() != \"Windows\":\n Popen('chmod og-rw ~/.netrc', shell=True)\nelse:\n # Copy dodsrc to working directory in Windows \n shutil.copy2(homeDir + '.dodsrc', os.getcwd())\n print('Copied .dodsrc to:', os.getcwd())" + "objectID": "how-tos/plot.html", + "href": "how-tos/plot.html", + "title": "How do I plot data?", + "section": "", + "text": "Coming Soon!" }, { - "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#finding-data-files-in-nasas-earthdata-cloud-or-whats-a-conceptid", - "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#finding-data-files-in-nasas-earthdata-cloud-or-whats-a-conceptid", - "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", - "section": "3. Finding Data Files in NASA’s EarthData Cloud: or What’s a ConceptID?", - "text": "3. Finding Data Files in NASA’s EarthData Cloud: or What’s a ConceptID?\nCloud OPeNDAP-enabled granules are organized b a unique identifier called the Concept ID. Below, we’ll demonstrate how to use NASA’s CMR API and either the Daymet dataset DOI or the shortname to obtain the Concept ID for Daymet Daily data. With this information, we can search all of NASA’s Earthdata holdings for matching data files (granules) based on our spatial area and time period of interest.\nSearch NASA Earthdata Holdings based on… * DOI - The Daymet daily data Digital Object Identifier * ConceptID - remember, this is NASA Earthdata’s unique ID for this dataset * Area of Interest * Time Range of Interest" + "objectID": "how-tos/plot.html#how-do-i-plot-a-netcdf-variable", + "href": "how-tos/plot.html#how-do-i-plot-a-netcdf-variable", + "title": "How do I plot data?", + "section": "How do I plot a NetCDF variable?", + "text": "How do I plot a NetCDF variable?" }, { - "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#obtaining-the-daymet-v4-daily-data-concept-id", - "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#obtaining-the-daymet-v4-daily-data-concept-id", - "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", - "section": "3.1 Obtaining the Daymet V4 Daily data Concept ID", - "text": "3.1 Obtaining the Daymet V4 Daily data Concept ID\nUsing Python’s requests library, we will build and provide the URL doisearch to search NASA’s Common Metadata Repository (CMR) to acquire the concept_id unique to the Daymet V4 Daily data in NASA’s Cloud holdings. We’ll use the concept_id in a next step when we search NASA Cloud Holdings for our spatial and temporal area of interest.\n\n# CMR API base url\ncmrurl='https://cmr.earthdata.nasa.gov/search/' # define the base url of NASA's CMR API as the variable `cmrurl`\ndoi = '10.3334/ORNLDAAC/2129' # Daymet V4R1 DOI - define the DOI as the variable `daymet_doi`\ndoisearch = cmrurl + 'collections.json?doi=' + doi\nprint(doisearch)\n\nconcept_id = requests.get(doisearch).json()['feed']['entry'][0]['id']\nprint(concept_id)\n\nhttps://cmr.earthdata.nasa.gov/search/collections.json?doi=10.3334/ORNLDAAC/2129\nC2532426483-ORNL_CLOUD" + "objectID": "how-tos/reformat.html", + "href": "how-tos/reformat.html", + "title": "Reformat data files", + "section": "", + "text": "Upcoming!" }, { - "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#searching-for-daymet-files-using-the-cmr-search-api", - "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#searching-for-daymet-files-using-the-cmr-search-api", - "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", - "section": "4. Searching for Daymet Files using the CMR Search API", - "text": "4. Searching for Daymet Files using the CMR Search API\nDaymet daily data files (or granules) are in netCDF4 format, and each file has one year’s worth of data. Data files are organized by variables (each for dayl, prcp, tmin, tmax, srad, swe, vp) and regions (each for us, pr, hi). Daymet filenames can be used to identify the files from continental North America (*_na_*.nc). The files from Puerto Rico and Hawaii are named as (*_pr_*.nc) and (*_hi_*.nc) respectively.\nBelow, we create appropriately formatted strings for our temporal range:\n\nstart_date = dt.datetime(2010, 1, 1) # specify your own start date\nend_date = dt.datetime(2011, 12, 31) # specify your end start date\nprint(start_date,end_date,sep='\\n')\n\ndt_format = '%Y-%m-%dT%H:%M:%SZ' # format requirement for datetime search\ntemporal_str = start_date.strftime(dt_format) + ',' + end_date.strftime(dt_format)\nprint(temporal_str)\n\n2010-01-01 00:00:00\n2011-12-31 00:00:00\n2010-01-01T00:00:00Z,2011-12-31T00:00:00Z\n\n\nWe will first search all the granules for the full extent, and time period of interest (2010, 2011). For this tutorial, we set up the search parameters and download maximum temperature data (tmax).\n\ncmr_url = 'https://cmr.earthdata.nasa.gov/search/granules'\n\ncmr_response = requests.get(cmr_url, \n params={\n 'concept_id': concept_id,\n 'temporal': temporal_str,\n 'page_size': 200,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\n\ndaymet_granules = cmr_response.json()['feed']['entry']\n\ndaymet_granules_urls = []\n\nfor granule in daymet_granules:\n item = next((item['href'] for item in granule['links'] if \"opendap\" in item[\"href\"] and \"tmax\" in item[\"href\"]), None)\n if item != None:\n daymet_granules_urls.append(item)\n\ndaymet_granules_urls\n\n['https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_hi_tmax_2010.nc',\n 'https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_na_tmax_2010.nc',\n 'https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_pr_tmax_2010.nc',\n 'https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_hi_tmax_2011.nc',\n 'https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_pr_tmax_2011.nc',\n 'https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_na_tmax_2011.nc']" + "objectID": "how-tos/reformat.html#netcdf-to-zarr", + "href": "how-tos/reformat.html#netcdf-to-zarr", + "title": "Reformat data files", + "section": "NetCDF to Zarr", + "text": "NetCDF to Zarr" }, { - "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#open-and-subset-granules-using-netcdf4-python-and-the-dap4-protocol", - "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#open-and-subset-granules-using-netcdf4-python-and-the-dap4-protocol", - "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", - "section": "5. Open and Subset Granules Using netcdf4-python and the DAP4 Protocol", - "text": "5. Open and Subset Granules Using netcdf4-python and the DAP4 Protocol\nThe “DAP4” protocol is included with Cloud OPeNDAP-enabled granules. This allows for certain granules to have their variables organized into group hierarchies, complex variable names retrieved, and to further distinguish dataset variables between each other. Because of this difference over on-premises OPeNDAP, which used DAP2, certain programming libraries may require updated methods for accessing Cloud OPeNDAP-enabled granules.\nIt is required to use the dap4:// protocol in the URL instead of https:// for the netcdf4-python library. Before the first granule in the list is accessed, we first replace the URL protocol in the string, then view its metadata using the netcdf4-python library.\nNOTE: Occasionally, due to server load, “BES Connection” errors may appear while opening a Cloud OPeNDAP granule. These errors do not affect granule access.\n\n#dap4_url = daymet_granules_urls.replace(\"https://\", \"dap4://\")\ndap4_url = [url.replace(\"https://\", \"dap4://\") for url in daymet_granules_urls]\n\nf = nc4.Dataset(dap4_url[0], 'r') # HI in 2010\nf\n\n<class 'netCDF4._netCDF4.Dataset'>\nroot group (NETCDF4 data model, file format DAP4):\n start_year: 2010\n source: Daymet Software Version 4.0\n Version_software: Daymet Software Version 4.0\n Version_data: Daymet Data Version 4.0\n Conventions: CF-1.6\n citation: Please see http://daymet.ornl.gov/ for current Daymet data citation information\n references: Please see http://daymet.ornl.gov/ for current information on Daymet references\n dimensions(sizes): y(584), x(284), time(365), nv(2)\n variables(dimensions): float32 y(y), float32 lon(y, x), float32 lat(y, x), float32 time(time), float32 x(x), float32 tmax(time, y, x), int16 lambert_conformal_conic(), float32 time_bnds(time, nv), int16 yearday(time)\n groups:" + "objectID": "how-tos/reformat.html#geotiff-to-cog", + "href": "how-tos/reformat.html#geotiff-to-cog", + "title": "Reformat data files", + "section": "GeoTIFF to COG", + "text": "GeoTIFF to COG" }, { - "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#open-and-subset-granules-using-xarray", - "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#open-and-subset-granules-using-xarray", - "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", - "section": "6. Open and Subset Granules Using Xarray", - "text": "6. Open and Subset Granules Using Xarray\nXarray is a commonly-used and widely supported Python library used for accessing and analyzing remotely-hosted datasets. Below, we use the open_dataset() function to access our first Cloud OPeNDAP Daymet granule, and to view its metadata. Both the dap4:// and https:// protocols may be passed into the Xarray library, and subsetting must be done using Xarray.\n\nds = xr.open_dataset(daymet_granules_urls[0]) # HI in 2010\nds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (y: 584, x: 284, time: 365, nv: 2)\nCoordinates:\n * y (y) float32 -3.9e+04 -4e+04 ... -6.21e+05 -6.22e+05\n lon (y, x) float32 ...\n lat (y, x) float32 ...\n * time (time) datetime64[ns] 2010-01-01T12:00:00 ... 20...\n * x (x) float32 -5.802e+06 -5.801e+06 ... -5.519e+06\nDimensions without coordinates: nv\nData variables:\n tmax (time, y, x) float32 ...\n lambert_conformal_conic int16 -32767\n time_bnds (time, nv) datetime64[ns] 2010-01-01 ... 2011-01-01\n yearday (time) int16 1 2 3 4 5 6 ... 361 362 363 364 365\nAttributes:\n start_year: 2010\n source: Daymet Software Version 4.0\n Version_software: Daymet Software Version 4.0\n Version_data: Daymet Data Version 4.0\n Conventions: CF-1.6\n citation: Please see http://daymet.ornl.gov/ f...\n references: Please see http://daymet.ornl.gov/ f...\n build_dmrpp_metadata.build_dmrpp: 3.20.13-563\n build_dmrpp_metadata.bes: 3.20.13-563\n build_dmrpp_metadata.libdap: libdap-3.20.11-193\n build_dmrpp_metadata.configuration: \\n# TheBESKeys::get_as_config()\\nAll...\n build_dmrpp_metadata.invocation: build_dmrpp -c /tmp/bes_conf_p6Fo -f...xarray.DatasetDimensions:y: 584x: 284time: 365nv: 2Coordinates: (5)y(y)float32-3.9e+04 -4e+04 ... -6.22e+05units :mlong_name :y coordinate of projectionstandard_name :projection_y_coordinatearray([ -39000., -40000., -41000., ..., -620000., -621000., -622000.],\n dtype=float32)lon(y, x)float32...units :degrees_eastlong_name :longitude coordinatestandard_name :longitude[165856 values with dtype=float32]lat(y, x)float32...units :degrees_northlong_name :latitude coordinatestandard_name :latitude[165856 values with dtype=float32]time(time)datetime64[ns]2010-01-01T12:00:00 ... 2010-12-...standard_name :timebounds :time_bndslong_name :24-hour day based on local timearray(['2010-01-01T12:00:00.000000000', '2010-01-02T12:00:00.000000000',\n '2010-01-03T12:00:00.000000000', ..., '2010-12-29T12:00:00.000000000',\n '2010-12-30T12:00:00.000000000', '2010-12-31T12:00:00.000000000'],\n dtype='datetime64[ns]')x(x)float32-5.802e+06 ... -5.519e+06units :mlong_name :x coordinate of projectionstandard_name :projection_x_coordinatearray([-5802250., -5801250., -5800250., ..., -5521250., -5520250., -5519250.],\n dtype=float32)Data variables: (4)tmax(time, y, x)float32...long_name :daily maximum temperatureunits :degrees Cgrid_mapping :lambert_conformal_coniccell_methods :area: mean time: maximum[60537440 values with dtype=float32]lambert_conformal_conic()int16...grid_mapping_name :lambert_conformal_coniclongitude_of_central_meridian :-100.0latitude_of_projection_origin :42.5false_easting :0.0false_northing :0.0standard_parallel :[25. 60.]semi_major_axis :6378137.0inverse_flattening :298.257223563array(-32767, dtype=int16)time_bnds(time, nv)datetime64[ns]...array([['2010-01-01T00:00:00.000000000', '2010-01-02T00:00:00.000000000'],\n ['2010-01-02T00:00:00.000000000', '2010-01-03T00:00:00.000000000'],\n ['2010-01-03T00:00:00.000000000', '2010-01-04T00:00:00.000000000'],\n ...,\n ['2010-12-29T00:00:00.000000000', '2010-12-30T00:00:00.000000000'],\n ['2010-12-30T00:00:00.000000000', '2010-12-31T00:00:00.000000000'],\n ['2010-12-31T00:00:00.000000000', '2011-01-01T00:00:00.000000000']],\n dtype='datetime64[ns]')yearday(time)int16...long_name :day of year (DOY) starting with day 1 on Januaray 1starray([ 1, 2, 3, ..., 363, 364, 365], dtype=int16)Attributes: (12)start_year :2010source :Daymet Software Version 4.0Version_software :Daymet Software Version 4.0Version_data :Daymet Data Version 4.0Conventions :CF-1.6citation :Please see http://daymet.ornl.gov/ for current Daymet data citation informationreferences :Please see http://daymet.ornl.gov/ for current information on Daymet referencesbuild_dmrpp_metadata.build_dmrpp :3.20.13-563build_dmrpp_metadata.bes :3.20.13-563build_dmrpp_metadata.libdap :libdap-3.20.11-193build_dmrpp_metadata.configuration :\n# TheBESKeys::get_as_config()\nAllowedHosts=^https?:\\/\\/\nBES.Catalog.catalog.FollowSymLinks=Yes\nBES.Catalog.catalog.RootDirectory=/tmp/tmp9fn76qqj/\nBES.Catalog.catalog.TypeMatch=dmrpp:.*\\.(dmrpp)$;\nBES.Catalog.catalog.TypeMatch+=h5:.*(\\.bz2|\\.gz|\\.Z)?$;\nBES.Data.RootDirectory=/dev/null\nBES.LogName=./bes.log\nBES.UncompressCache.dir=/tmp/hyrax_ux\nBES.UncompressCache.prefix=ux_\nBES.UncompressCache.size=500\nBES.module.cmd=/usr/lib64/bes/libdap_xml_module.so\nBES.module.dap=/usr/lib64/bes/libdap_module.so\nBES.module.dmrpp=/usr/lib64/bes/libdmrpp_module.so\nBES.module.fonc=/usr/lib64/bes/libfonc_module.so\nBES.module.h5=/usr/lib64/bes/libhdf5_module.so\nBES.module.nc=/usr/lib64/bes/libnc_module.so\nBES.modules=dap,cmd,h5,dmrpp,nc,fonc\nFONc.ClassicModel=false\nFONc.NoGlobalAttrs=true\nH5.EnableCF=false\nH5.EnableCheckNameClashing=true\nbuild_dmrpp_metadata.invocation :build_dmrpp -c /tmp/bes_conf_p6Fo -f /tmp/tmp9fn76qqj//daymet_v4_daily_hi_tmax_2010.nc -r /tmp/dmr__CDJLKz -u OPeNDAP_DMRpp_DATA_ACCESS_URL -M" + "objectID": "how-tos/aggregate.html", + "href": "how-tos/aggregate.html", + "title": "Aggregate data", + "section": "", + "text": "Coming soon!" }, { - "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#resample-and-plot-tmax", - "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#resample-and-plot-tmax", - "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", - "section": "7. Resample and Plot tmax", - "text": "7. Resample and Plot tmax\nBelow, we will resample the tmax variable and calculate the monthly mean using Xarray’s built-in functions. Then, we will plot the monthly tmax mean for the month of July in Hawaii for 2010.\n\n# Monthly resample\nmonthly_tmax_mean = ds['tmax'].resample(time=\"M\").mean()\nmonthly_tmax_mean\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray 'tmax' (time: 12, y: 584, x: 284)>\narray([[[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]],\n\n [[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]],\n\n [[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n...\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]],\n\n [[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]],\n\n [[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]]], dtype=float32)\nCoordinates:\n * time (time) datetime64[ns] 2010-01-31 2010-02-28 ... 2010-12-31\n * y (y) float32 -3.9e+04 -4e+04 -4.1e+04 ... -6.21e+05 -6.22e+05\n lon (y, x) float32 ...\n lat (y, x) float32 ...\n * x (x) float32 -5.802e+06 -5.801e+06 -5.8e+06 ... -5.52e+06 -5.519e+06xarray.DataArray'tmax'time: 12y: 584x: 284nan nan nan nan nan nan nan nan ... nan nan nan nan nan nan nan nanarray([[[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]],\n\n [[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]],\n\n [[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n...\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]],\n\n [[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]],\n\n [[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]]], dtype=float32)Coordinates: (5)time(time)datetime64[ns]2010-01-31 ... 2010-12-31array(['2010-01-31T00:00:00.000000000', '2010-02-28T00:00:00.000000000',\n '2010-03-31T00:00:00.000000000', '2010-04-30T00:00:00.000000000',\n '2010-05-31T00:00:00.000000000', '2010-06-30T00:00:00.000000000',\n '2010-07-31T00:00:00.000000000', '2010-08-31T00:00:00.000000000',\n '2010-09-30T00:00:00.000000000', '2010-10-31T00:00:00.000000000',\n '2010-11-30T00:00:00.000000000', '2010-12-31T00:00:00.000000000'],\n dtype='datetime64[ns]')y(y)float32-3.9e+04 -4e+04 ... -6.22e+05units :mlong_name :y coordinate of projectionstandard_name :projection_y_coordinatearray([ -39000., -40000., -41000., ..., -620000., -621000., -622000.],\n dtype=float32)lon(y, x)float32...units :degrees_eastlong_name :longitude coordinatestandard_name :longitude[165856 values with dtype=float32]lat(y, x)float32...units :degrees_northlong_name :latitude coordinatestandard_name :latitude[165856 values with dtype=float32]x(x)float32-5.802e+06 ... -5.519e+06units :mlong_name :x coordinate of projectionstandard_name :projection_x_coordinatearray([-5802250., -5801250., -5800250., ..., -5521250., -5520250., -5519250.],\n dtype=float32)Attributes: (0)\n\n\n\n# Xarray plotting\nmonthly_tmax_mean[6,:,:].plot()\n\n<matplotlib.collections.QuadMesh at 0x7fdaba88de50>" + "objectID": "how-tos/aggregate.html#how-do-i-calculate-a-climatology", + "href": "how-tos/aggregate.html#how-do-i-calculate-a-climatology", + "title": "Aggregate data", + "section": "How do I calculate a climatology?", + "text": "How do I calculate a climatology?" }, { - "objectID": "tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html", - "href": "tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html", - "title": "Hurricanes: Wind and Sea Surface Temperature", + "objectID": "in-development/earthdata-python-r-handoff.html", + "href": "in-development/earthdata-python-r-handoff.html", + "title": "earthdata: Python-R Handoff", "section": "", - "text": "This notebook demonstrates how to access two cloud-hosted Earthdata granules using the Commmon Metadata Repository (CMR) API. The granules are from two different DAACs (GES DISC and PO.DAAC). It shows the process of obtaining and plotting two variables from two distinct granules hosted in S3 buckets: sea surface temperature (SST) from the GHRSST Level 4 MUR Global Foundation Sea Surface Temperature Analysis (v4.1) (GHRSST)) and calculated 2-meter (2M) wind velocity from the MERRA-2 hourly time-averaged reanalysis dataset (M2T1NXSLV.5.12.4), before plotting them together on 25 October 2012 over the Caribbean Sea when Hurricane Sandy was at peak strength." + "text": "Create once, use often: using earthdata python package for NASA Earthdata authorization and identifying the s3 links (i.e. the locations where the data are stored on Amazon Web Services), then passing those python objects to R through Quarto for analysis by R folks. These notes are a work-in-progress by Julie and Luis and we’ll tidy them up as we develop them further.\nNote: this dream is currently not working but we are sharing our progress." }, { - "objectID": "tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html#overview", - "href": "tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html#overview", - "title": "Hurricanes: Wind and Sea Surface Temperature", + "objectID": "in-development/earthdata-python-r-handoff.html#the-dream", + "href": "in-development/earthdata-python-r-handoff.html#the-dream", + "title": "earthdata: Python-R Handoff", "section": "", - "text": "This notebook demonstrates how to access two cloud-hosted Earthdata granules using the Commmon Metadata Repository (CMR) API. The granules are from two different DAACs (GES DISC and PO.DAAC). It shows the process of obtaining and plotting two variables from two distinct granules hosted in S3 buckets: sea surface temperature (SST) from the GHRSST Level 4 MUR Global Foundation Sea Surface Temperature Analysis (v4.1) (GHRSST)) and calculated 2-meter (2M) wind velocity from the MERRA-2 hourly time-averaged reanalysis dataset (M2T1NXSLV.5.12.4), before plotting them together on 25 October 2012 over the Caribbean Sea when Hurricane Sandy was at peak strength." + "text": "Create once, use often: using earthdata python package for NASA Earthdata authorization and identifying the s3 links (i.e. the locations where the data are stored on Amazon Web Services), then passing those python objects to R through Quarto for analysis by R folks. These notes are a work-in-progress by Julie and Luis and we’ll tidy them up as we develop them further.\nNote: this dream is currently not working but we are sharing our progress." }, { - "objectID": "tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html#learning-objectives", - "href": "tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html#learning-objectives", - "title": "Hurricanes: Wind and Sea Surface Temperature", - "section": "Learning Objectives", - "text": "Learning Objectives\n\nSearch for data programmatically using known parameters\nAccess data within the AWS cloud using the earthaccess python library\nAccess data archived by 2 NASA data centers from the AWS cloud\nVisualize wind speed and sea surface temperature together during a hurricane event" + "objectID": "in-development/earthdata-python-r-handoff.html#python-earthdata-package-for-auth-s3-links", + "href": "in-development/earthdata-python-r-handoff.html#python-earthdata-package-for-auth-s3-links", + "title": "earthdata: Python-R Handoff", + "section": "Python: earthdata package for auth & s3 links", + "text": "Python: earthdata package for auth & s3 links\nearthdata gets me the credentials, it gets me the links based on the queries.\nIn this example, the data we want is in the Cloud. For this examples we’re using this data we identified from the Earthdata Cloud Cookbook’s Multi-File_Direct_S3_Access_NetCDF_Example, and its short_name is 'ECCO_L4_SSH_05DEG_MONTHLY_V4R4'.\n\nIdentify the s3 links\nBelow is our query, pretending that that is the data and the bounding box we want.\n\n## import DataCollections class from earthdata library\nfrom earthdata import DataGranules\n\n## To find the concept_id from the shortname that we copied: \n# short_name = 'ECCO_L4_SSH_05DEG_MONTHLY_V4R4' \n# collection = DataCollections().short_name(short_name).get()\n# [c.concept_id() for c in collection] ## this returned 'C1990404799-POCLOUD'\n\n# Then we build a Query with spatiotemporal parameters. \nGranuleQuery = DataGranules().concept_id('C1990404799-POCLOUD').bounding_box(-134.7,58.9,-133.9,59.2)\n\n## We get the metadata records from CMR\ngranules = GranuleQuery.get()\n\n## Now it's time to open our data granules list. \ns3_links = [granule.data_links(access='direct') for granule in granules] \ns3_links[0]\n\nNote that files = Store(auth).open(granules) would work for Python users but open won’t work in the R world because it will create some kind of python file handlers from fsspec.\n\n\nGet the Cloud credentials\nPrerequesite: you’ll need a functioning .netrc here. earthdata expects interactivity and that did not work here with Quarto in the RStudio IDE (and it also did not work for Julie in Jupyter notebook (June 7 2022)). So, we followed the 2021-Cloud-Hackathon’s NASA_Earthdata_Authentication, copying and pasting and running that code in a Jupyter notebook. (remember to rm .netrc beforehand!)\nThen, with a nice .netrc file, the next step is to get Cloud credentials:\n\n## import the Auth class from the earthdata library\nfrom earthdata import Auth\n\nauth = Auth().login(strategy=\"netrc\")\ncredentials = auth.get_s3_credentials(cloud_provider = \"POCLOUD\") \n\nSo now we have the s3 links and the credentials to download the links, so now we can use the tutorial in R!!\nNotes\n\nLuis will update earthdata to automatically know the cloud provider so that you don’t have to specify for example POCLOUD vs PODAAC # credentials you actually don’t want to print your credentials, we were just checking that they worked\nThe resulting JSON dictionary is what we’ll export to R, and it will be valid for 1 hour. When I run into issues, I’ll say “why is this not working”, and it’s because it’s expired in 1 hour.\nWhen we want to identify the bucket level, we’ll need to remove the name of the file. For example:\n\n<s3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_1992-01_ECCO_V4r4_latlon_0p50deg.nc> includes the filename\n<s3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/> is only the bucket\n\nExpect to run into issues with listing the files in the bucket (because maybe something is restricted or maybe you can access files but not list everything that’s inside the bucket)" }, { - "objectID": "tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html#requirements", - "href": "tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html#requirements", - "title": "Hurricanes: Wind and Sea Surface Temperature", - "section": "Requirements", - "text": "Requirements\n\n1. Compute environment - This notebook can only be run in an AWS Cloud instance running in the us-west-2 region.\n\n\n2. Earthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\nInstall Packages\n\nimport earthaccess\nimport xarray as xr\nimport requests\nimport s3fs\nimport numpy as np\nimport pprint\nimport cartopy.crs as ccrs\nimport cartopy.feature as cfeature\nimport matplotlib.pyplot as plt\nimport boto3\nfrom IPython.display import display, Markdown\n\n%matplotlib inline\n\nWe recommend authenticating your Earthdata Login (EDL) information using the earthaccess python package as follows:\n\nauth = earthaccess.login(strategy=\"netrc\") # works if the EDL login already been persisted to a netrc\nif not auth.authenticated:\n # ask for EDL credentials and persist them in a .netrc file\n auth = earthaccess.login(strategy=\"interactive\", persist=True)\n\nYou're now authenticated with NASA Earthdata Login\nearthaccess generated a token for CMR with expiration on: 10/29/2023\nearthaccess generated a token for CMR with expiration on: 10/29/2023\nUsing .netrc file for EDL\n\n\n\n\nRetrieve Granule S3 Links\nTo retrieve the S3 URLs, we will use the CMR Search API to pass the “Short Name” of the dataset, the version of the dataset, the temporal range, and the page size for the JSON response. For GES DISC datasets, you can find the short name and version by navigating to any dataset landing page, where it will be listed in the “Product Summary” tab. We will also retrieve the S3 token endpoints, used for accessing the S3 buckets with our Earthdata credentials:\n\ncmr_url = 'https://cmr.earthdata.nasa.gov/search/granules'\n\nmerra_response = requests.get(cmr_url, \n params={\n 'short_name': 'M2T1NXSLV',\n 'version': '5.12.4',\n 'temporal': '2012-10-25T00:00:00Z',\n 'page_size': 1,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\n\n# Parse S3 Token Endpoint and S3 URL\n\nprint('S3 Token Endpoint:')\n\nfor element in merra_response.json()['feed']['entry'][0]['links']:\n if element['href'].endswith('s3credentials'):\n print(element['href'])\n gesdisc_s3 = element['href']\n \nprint('S3 URL:')\n\nfor element in merra_response.json()['feed']['entry'][0]['links']:\n if element['rel'] == 'http://esipfed.org/ns/fedsearch/1.1/s3#':\n print(element['href'])\n merra_fn = element['href']\n\nS3 Token Endpoint:\nhttps://data.gesdisc.earthdata.nasa.gov/s3credentials\nS3 URL:\ns3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2012/10/MERRA2_400.tavg1_2d_slv_Nx.20121025.nc4\n\n\nThen, we build a similar request for retrieving our sea surface temperature data from the GHRSST dataset at 2012-10-25 00Z. The short name and version for this dataset was retrieved from its PO.DAAC dataset landing page, under the “Information” tab.\n\nghrsst_response = requests.get(cmr_url, \n params={\n 'short_name': 'MUR-JPL-L4-GLOB-v4.1',\n 'version': '4.1',\n 'temporal': '2012-10-25T00:00:00Z',\n 'page_size': 1,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\n\n# Parse S3 Token Endpoint and S3 URL\n\nprint('S3 Token Endpoint:')\n\nfor element in ghrsst_response.json()['feed']['entry'][0]['links']:\n if element['href'].endswith('s3credentials'):\n print(element['href'])\n podaac_s3 = element['href']\n \nprint('S3 URL:')\n\nfor element in ghrsst_response.json()['feed']['entry'][0]['links']:\n if element['rel'] == 'http://esipfed.org/ns/fedsearch/1.1/s3#':\n print(element['href'])\n ghrsst_fn = element['href']\n\nS3 Token Endpoint:\nhttps://archive.podaac.earthdata.nasa.gov/s3credentials\nS3 URL:\ns3://podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20121025090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc\n\n\n\n\nObtain S3 Credentials and Open Bucket Granules\nDirect S3 access is granted to the user through a temporary token, which will last for one hour, and will need to be rerun after that hour has passed to access the bucket. Then, to access the bucket, we can pass this token into the S3FS S3FileSystem() function which will “mount” the S3 bucket to our notebook as if it were a locally stored file system. Here, we create a function that queries the S3 Token Endpoint with their Earthdata credentials, stored in a netrc file, using the Python requests library. Next, it returns an s3fs.core.S3FileSystem object that represents the “mounted” S3 bucket:\n\n# Define a function for S3 access credentials\n\ndef begin_s3_direct_access(daac_url):\n \n # Retrieve the token as a JSON\n response = requests.get(daac_url).json()\n \n # Mount the bucket and return it as an S3FileSystem object\n return s3fs.S3FileSystem(key=response['accessKeyId'],\n secret=response['secretAccessKey'],\n token=response['sessionToken'],\n client_kwargs={'region_name':'us-west-2'})\n\nWe need two tokens in order to access the two different buckets, so we will perform two function calls with the two S3 Token Endpoints retrieved from CMR, stored in separate variables. If you immediately receive an error, double-check that your username and password were entered correctly in your netrc file, or that you can access the following URL: https://data.gesdisc.earthdata.nasa.gov\n\n# Open S3 file systems with S3FS\n\ngesdisc_fs = begin_s3_direct_access(gesdisc_s3)\npodaac_fs = begin_s3_direct_access(podaac_s3)\n\n# Check that the file system is intact as an S3FileSystem object, which means that token is valid\n\ntype(gesdisc_fs)\ntype(podaac_fs)\n\ns3fs.core.S3FileSystem\n\n\n\n\nOpen Granules in Xarray\nIn order to open the granules in Xarray, we must use the S3FS open() function. Here, we pass the S3 URLs from each separate file system into two different variables:\n\n# Open datasets with S3FS\n\nmerra_ds = xr.open_dataset(gesdisc_fs.open(merra_fn))\nghrsst_ds = xr.open_dataset(podaac_fs.open(ghrsst_fn))\n\nNow, the granules are stored in memory as Xarray datasets, which will be stored as long as the kernel is running. Here, we clip to the extent over where the hurricane was:\n\nmin_lon = -89\nmin_lat = 14\nmax_lon = -67\nmax_lat = 31\n\nmerra_ds = merra_ds.sel(lat=slice(min_lat,max_lat), lon=slice(min_lon,max_lon))\nghrsst_ds = ghrsst_ds.sel(lat=slice(min_lat,max_lat), lon=slice(min_lon,max_lon))\n\n\n\nConvert Dataset Grids\nHere, we interpolate the GHRSST grid to the MERRA grid using Xarray’s interp() function:\n\nghrsst_ds = ghrsst_ds.interp(lat=merra_ds.lat, lon=merra_ds.lon)\nghrsst_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (time: 1, lat: 35, lon: 35)\nCoordinates:\n * time (time) datetime64[ns] 2012-10-25T09:00:00\n * lat (lat) float64 14.0 14.5 15.0 15.5 ... 29.5 30.0 30.5 31.0\n * lon (lon) float64 -88.75 -88.12 -87.5 ... -68.75 -68.12 -67.5\nData variables:\n analysed_sst (time, lat, lon) float64 nan nan nan ... 300.0 299.4 299.4\n analysis_error (time, lat, lon) float64 nan nan nan ... 0.37 0.37 0.37\n mask (time, lat, lon) float64 2.0 2.0 2.0 2.0 ... 1.0 1.0 1.0\n sea_ice_fraction (time, lat, lon) float64 nan nan nan nan ... nan nan nan\nAttributes: (12/47)\n Conventions: CF-1.5\n title: Daily MUR SST, Final product\n summary: A merged, multi-sensor L4 Foundation SST anal...\n references: http://podaac.jpl.nasa.gov/Multi-scale_Ultra-...\n institution: Jet Propulsion Laboratory\n history: created at nominal 4-day latency; replaced nr...\n ... ...\n project: NASA Making Earth Science Data Records for Us...\n publisher_name: GHRSST Project Office\n publisher_url: http://www.ghrsst.org\n publisher_email: ghrsst-po@nceo.ac.uk\n processing_level: L4\n cdm_data_type: gridxarray.DatasetDimensions:time: 1lat: 35lon: 35Coordinates: (3)time(time)datetime64[ns]2012-10-25T09:00:00long_name :reference time of sst fieldstandard_name :timeaxis :Tcomment :Nominal time of analyzed fieldsarray(['2012-10-25T09:00:00.000000000'], dtype='datetime64[ns]')lat(lat)float6414.0 14.5 15.0 ... 30.0 30.5 31.0long_name :latitudeunits :degrees_northvmax :1000000000000000.0vmin :-1000000000000000.0valid_range :[-1.e+15 1.e+15]array([14. , 14.5, 15. , 15.5, 16. , 16.5, 17. , 17.5, 18. , 18.5, 19. , 19.5,\n 20. , 20.5, 21. , 21.5, 22. , 22.5, 23. , 23.5, 24. , 24.5, 25. , 25.5,\n 26. , 26.5, 27. , 27.5, 28. , 28.5, 29. , 29.5, 30. , 30.5, 31. ])lon(lon)float64-88.75 -88.12 ... -68.12 -67.5long_name :longitudeunits :degrees_eastvmax :1000000000000000.0vmin :-1000000000000000.0valid_range :[-1.e+15 1.e+15]array([-88.75 , -88.125, -87.5 , -86.875, -86.25 , -85.625, -85. , -84.375,\n -83.75 , -83.125, -82.5 , -81.875, -81.25 , -80.625, -80. , -79.375,\n -78.75 , -78.125, -77.5 , -76.875, -76.25 , -75.625, -75. , -74.375,\n -73.75 , -73.125, -72.5 , -71.875, -71.25 , -70.625, -70. , -69.375,\n -68.75 , -68.125, -67.5 ])Data variables: (4)analysed_sst(time, lat, lon)float64nan nan nan ... 300.0 299.4 299.4long_name :analysed sea surface temperaturestandard_name :sea_surface_foundation_temperatureunits :kelvinvalid_min :-32767valid_max :32767comment :\"Final\" version using Multi-Resolution Variational Analysis (MRVA) method for interpolationsource :AVHRR18_G-NAVO, AVHRR19_G-NAVO, AVHRR_METOP_A-EUMETSAT, MODIS_A-JPL, MODIS_T-JPL, WSAT-REMSS, iQUAM-NOAA/NESDIS, Ice_Conc-OSISAFarray([[[ nan, nan, nan, ..., 302.08099365,\n 301.34849548, 302.09698486],\n [ nan, nan, nan, ..., 301.13699341,\n 301.77400208, 301.94799805],\n [ nan, nan, nan, ..., 302.04498291,\n 302.22299194, 302.26998901],\n ...,\n [297.48300171, 298.64399719, 298.88000488, ..., 300.36499023,\n 300.13198853, 299.65600586],\n [ nan, nan, nan, ..., 300.27600098,\n 299.01649475, 299.55300903],\n [ nan, nan, nan, ..., 300.01699829,\n 299.39898682, 299.44500733]]])analysis_error(time, lat, lon)float64nan nan nan nan ... 0.37 0.37 0.37long_name :estimated error standard deviation of analysed_sstunits :kelvinvalid_min :0valid_max :32767comment :nonearray([[[ nan, nan, nan, ..., 0.38 ,\n 0.38 , 0.38 ],\n [ nan, nan, nan, ..., 0.38 ,\n 0.38 , 0.38999999],\n [ nan, nan, nan, ..., 0.38999999,\n 0.38 , 0.39999998],\n ...,\n [0.37 , 0.37 , 0.37 , ..., 0.37 ,\n 0.37 , 0.37 ],\n [ nan, nan, nan, ..., 0.37 ,\n 0.38999999, 0.37 ],\n [ nan, nan, nan, ..., 0.37 ,\n 0.37 , 0.37 ]]])mask(time, lat, lon)float642.0 2.0 2.0 2.0 ... 1.0 1.0 1.0 1.0long_name :sea/land field composite maskvalid_min :1valid_max :31flag_masks :[ 1 2 4 8 16]flag_values :[ 1 2 5 9 13]flag_meanings :1=open-sea, 2=land, 5=open-lake, 9=open-sea with ice in the grid, 13=open-lake with ice in the gridcomment :mask can be used to further filter the data.source :GMT \"grdlandmask\", ice flag from sea_ice_fraction dataarray([[[2., 2., 2., ..., 1., 1., 1.],\n [2., 2., 2., ..., 1., 1., 1.],\n [2., 2., 2., ..., 1., 1., 1.],\n ...,\n [1., 1., 1., ..., 1., 1., 1.],\n [2., 2., 2., ..., 1., 1., 1.],\n [2., 2., 2., ..., 1., 1., 1.]]])sea_ice_fraction(time, lat, lon)float64nan nan nan nan ... nan nan nan nanlong_name :sea ice area fractionstandard_name :sea ice area fractionunits :fraction (between 0 and 1)valid_min :0valid_max :100source :EUMETSAT OSI-SAF, copyright EUMETSATcomment :ice data interpolated by a nearest neighbor approach.array([[[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]]])Indexes: (3)timePandasIndexPandasIndex(DatetimeIndex(['2012-10-25 09:00:00'], dtype='datetime64[ns]', name='time', freq=None))latPandasIndexPandasIndex(Index([14.0, 14.5, 15.0, 15.5, 16.0, 16.5, 17.0, 17.5, 18.0, 18.5, 19.0, 19.5,\n 20.0, 20.5, 21.0, 21.5, 22.0, 22.5, 23.0, 23.5, 24.0, 24.5, 25.0, 25.5,\n 26.0, 26.5, 27.0, 27.5, 28.0, 28.5, 29.0, 29.5, 30.0, 30.5, 31.0],\n dtype='float64', name='lat'))lonPandasIndexPandasIndex(Index([ -88.75, -88.125, -87.5, -86.875, -86.25, -85.625, -85.0, -84.375,\n -83.75, -83.125, -82.5, -81.875, -81.25, -80.625, -80.0, -79.375,\n -78.75, -78.125, -77.5, -76.875, -76.25, -75.625, -75.0, -74.375,\n -73.75, -73.125, -72.5, -71.875, -71.25, -70.625, -70.0, -69.375,\n -68.75, -68.125, -67.5],\n dtype='float64', name='lon'))Attributes: (47)Conventions :CF-1.5title :Daily MUR SST, Final productsummary :A merged, multi-sensor L4 Foundation SST analysis product from JPL.references :http://podaac.jpl.nasa.gov/Multi-scale_Ultra-high_Resolution_MUR-SSTinstitution :Jet Propulsion Laboratoryhistory :created at nominal 4-day latency; replaced nrt (1-day latency) version.comment :MUR = \"Multi-scale Ultra-high Reolution\"license :These data are available free of charge under data policy of JPL PO.DAAC.id :MUR-JPL-L4-GLOB-v04.1naming_authority :org.ghrsstproduct_version :04.1uuid :27665bc0-d5fc-11e1-9b23-0800200c9a66gds_version_id :2.0netcdf_version_id :4.1date_created :20150710T184006Zstart_time :20121025T090000Zstop_time :20121025T090000Ztime_coverage_start :20121024T210000Ztime_coverage_end :20121025T210000Zfile_quality_level :1source :AVHRR18_G-NAVO, AVHRR19_G-NAVO, AVHRR_METOP_A-EUMETSAT, MODIS_A-JPL, MODIS_T-JPL, WSAT-REMSS, iQUAM-NOAA/NESDIS, Ice_Conc-OSISAFplatform :Aqua, DMSP, NOAA-POES, Suomi-NPP, Terrasensor :AMSR-E, AVHRR, MODIS, SSM/I, VIIRS, in-situMetadata_Conventions :Unidata Observation Dataset v1.0metadata_link :http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MUR-JPL-L4-GLOB-v04.1keywords :Oceans > Ocean Temperature > Sea Surface Temperaturekeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordsstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventionsouthernmost_latitude :-90.0northernmost_latitude :90.0westernmost_longitude :-180.0easternmost_longitude :180.0spatial_resolution :0.01 degreesgeospatial_lat_units :degrees northgeospatial_lat_resolution :0.01 degreesgeospatial_lon_units :degrees eastgeospatial_lon_resolution :0.01 degreesacknowledgment :Please acknowledge the use of these data with the following statement: These data were provided by JPL under support by NASA MEaSUREs program.creator_name :JPL MUR SST projectcreator_email :ghrsst@podaac.jpl.nasa.govcreator_url :http://mur.jpl.nasa.govproject :NASA Making Earth Science Data Records for Use in Research Environments (MEaSUREs) Programpublisher_name :GHRSST Project Officepublisher_url :http://www.ghrsst.orgpublisher_email :ghrsst-po@nceo.ac.ukprocessing_level :L4cdm_data_type :grid\n\n\n\n\nPlot Variables using Matplotlib and Cartopy\nFinally, we use Matplotlib and Cartopy to plot a color mesh and quiver plot of SSTs and 2M winds of Hurricane Sandy at her peak strength:\n\n# Pre-configure wind vector variables\n\nu=merra_ds.U2M\nv=merra_ds.V2M\nlon=merra_ds.lon\nlat=merra_ds.lat\n\nlons, lats = np.meshgrid(lon, lat)\n\n# Plotting routines:\n\n# Figure size\nplt.rcParams['figure.figsize'] = 15, 15\n\n# Figure and geography setup\nfig = plt.figure()\nax = fig.add_subplot(2, 1, 1, projection=ccrs.PlateCarree())\nax.set_extent([-89, -67, 14, 31], crs=ccrs.PlateCarree())\nax.add_feature(cfeature.COASTLINE.with_scale('50m'), linewidth=0.5, zorder=5) \nax.add_feature(cfeature.LAND, facecolor='white', zorder=2) \nax.add_feature(cfeature.BORDERS, linewidth=0.5, zorder=5)\nax.add_feature(cfeature.STATES, zorder=5)\n\n# Colormesh of SSTs\nmmp = ax.pcolormesh(lons, lats, ghrsst_ds.analysed_sst.isel(time=0), \n cmap='hot_r', transform=ccrs.PlateCarree(), zorder=1)\n\n# Quiver plot of 2M vector field\nq = ax.quiver(lons, lats, u.isel(time=0).values, v.isel(time=0).values, zorder=4, \n transform=ccrs.PlateCarree(), scale_units='inches', color='gray')\n\n# Quiver key for scale\nax.quiverkey(q, 1.22, 1.05, 10, r'$10 \\frac{m}{s}$', zorder=4)\n\n# Lat/lon grid lines\nax.gridlines(draw_labels=True, dms=True, x_inline=False, y_inline=False)\n\n# SST color bar setup\ncbar = plt.colorbar(mmp, pad=0.1)\ncbar.set_label(\"Analyzed SST (K)\")\n\n# Figure title\nfig.suptitle(\"GHRSST Analyzed SST and MERRA-2 2M Wind Vectors on 2012-10-25T00:00:00Z\", size=16, y=0.95)\n\nText(0.5, 0.95, 'GHRSST Analyzed SST and MERRA-2 2M Wind Vectors on 2012-10-25T00:00:00Z')\n\n\n/srv/conda/envs/notebook/lib/python3.9/site-packages/cartopy/io/__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/50m_physical/ne_50m_land.zip\n warnings.warn(f'Downloading: {url}', DownloadWarning)\n/srv/conda/envs/notebook/lib/python3.9/site-packages/cartopy/io/__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/50m_cultural/ne_50m_admin_1_states_provinces_lakes.zip\n warnings.warn(f'Downloading: {url}', DownloadWarning)" + "objectID": "in-development/earthdata-python-r-handoff.html#r-data-access-from-s3-links", + "href": "in-development/earthdata-python-r-handoff.html#r-data-access-from-s3-links", + "title": "earthdata: Python-R Handoff", + "section": "R: data access from s3 links!", + "text": "R: data access from s3 links!\nAnd now I can switch to R, if R is my preferred language.\nThe blog post Using Amazon S3 with R by Danielle Navarro is hugely informative and describes how to use the aws.s3 R package.\nFirst load libraries:\n\nlibrary(dplyr)\nlibrary(readr)\nlibrary(purrr)\nlibrary(stringr)\nlibrary(tibble)\nlibrary(aws.s3) # install.packages(\"aws.s3\")\nlibrary(reticulate)\n\nTranslate credentials from python variables (created with earthdata above) to R variables using reticulate’s py$ syntax and purr’s pluck() to isolate a variable from a list:\n\n## translate credentials from python to R, map to dataframe\ncredentials_r_list <- py$credentials #YAY!\ncredentials_r <- purrr::map_df(credentials_r_list, print)\n\n## translate s3 links from python to R, create my_bucket\ns3_links_r_list <- py$s3_links\nmy_link_list <- s3_links_r_list[1] # let's just start with one\nmy_link_chr <- purrr:::map_chr(my_link_list, paste, collapse=\"\")\n#my_link <- as_tibble(my_link_chr)\n#my_link_split <- stringr::str_split(my_link, \"/\")\n#my_bucket <- str_c(\"s3://\", my_link_split[3], my_link_split[4])\nmy_bucket <- \"s3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/\"\n\nFrom the aws.s3 documentation, set up system environment variables for AWS:\n\nSys.setenv(\"AWS_ACCESS_KEY_ID\" = credentials_r$accessKeyId,\n \"AWS_SECRET_ACCESS_KEY\" = credentials_r$secretAccessKey,\n \"AWS_DEFAULT_REGION\" = \"us-west-2\",\n \"AWS_SESSION_TOKEN\" = credentials_r$sessionToken)\n\n\n# testing by hand: Luis\nSys.setenv(\"AWS_ACCESS_KEY_ID\" = \"ASIATNGJQBXBHRPIKFFB\",\n \"AWS_SECRET_ACCESS_KEY\" = \"zbYP2fueNxLK/joDAcz678mkjjzP6fz4HUN131ID\",\n \"AWS_DEFAULT_REGION\" = \"us-west-2\")\n\nFirst let’s test Danielle’s code to see if it runs. Note to Luis: the following only works when the Sys.setenv is not set:\n\nlibrary(aws.s3)\n\nbucket_exists(\n bucket = \"s3://herbariumnsw-pds/\", \n region = \"ap-southeast-2\"\n)\n\nClient error: (403) Forbidden\n\n\n[1] FALSE\nattr(,\"x-amz-bucket-region\")\n[1] \"ap-southeast-2\"\nattr(,\"x-amz-request-id\")\n[1] \"0FQ1R57F2VHGFPDF\"\nattr(,\"x-amz-id-2\")\n[1] \"N6RPTKPN3/H9tDuKNHM2ZAcChhkkn2WpfcTzhpxC3fUmiZdNEIiu1xJsQAvFSecYIuWZ28pchQW3sAPAdVU57Q==\"\nattr(,\"content-type\")\n[1] \"application/xml\"\nattr(,\"date\")\n[1] \"Thu, 07 Jul 2022 23:11:30 GMT\"\nattr(,\"server\")\n[1] \"AmazonS3\"\n\n\nNow, see if the PODAAC bucket exists:\n\naws.s3::bucket_exists(\n bucket = \"s3://podaac-ops-cumulus-protected/\", \n region = \"us-west-2\"\n)\n\nClient error: (403) Forbidden\n\n\n[1] FALSE\nattr(,\"x-amz-bucket-region\")\n[1] \"us-west-2\"\nattr(,\"x-amz-request-id\")\n[1] \"M4T3W1JZ93M08AZB\"\nattr(,\"x-amz-id-2\")\n[1] \"hvGLWqGCRB4lLf9pD8f67OsTDulSOgqd+yLWzUTRFz2tlLPVpxHr9mSREL0bQPVyo70j0hvJp+8=\"\nattr(,\"content-type\")\n[1] \"application/xml\"\nattr(,\"date\")\n[1] \"Thu, 07 Jul 2022 23:11:30 GMT\"\nattr(,\"server\")\n[1] \"AmazonS3\"\n\n\n\nherbarium_files <- get_bucket_df(\n bucket = \"s3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/\", \n region = \"us-west-2\",\n max = 20000\n) %>% \n as_tibble()\n\nIf forbidden: - 1 hour expiration time - this bucket is not listable (or protected) (hopefully this error will be clear enough)\nIf you get the following error, it’s likely because your credentials have expired:\n\n\n\n\n\n\nImportant\n\n\n\nClient error: (403) Forbidden\n[1] FALSE\nattr(,\"x-amz-bucket-region\")\n[1] \"us-west-2\"\nattr(,\"x-amz-request-id\")\n[1] \"W2PQV030PDTGDD32\"\nattr(,\"x-amz-id-2\")\n[1] \"S8C0qzL1lAYLufzUupjqplyyS/3fWCKxIELk0OJLVHGzTOqlyhof+IPFYbaRUhmJwXQelfprYCU=\"\nattr(,\"content-type\")\n[1] \"application/xml\"\nattr(,\"date\")\n[1] \"Wed, 08 Jun 2022 03:11:16 GMT\"\nattr(,\"server\")\n[1] \"AmazonS3\"" }, { - "objectID": "tutorials/Data_stories_template.html", - "href": "tutorials/Data_stories_template.html", - "title": "Title", - "section": "", - "text": "Add Summary of data story/background information needed. 2-5 sentences." + "objectID": "in-development/earthdata-python-r-handoff.html#dev-notes", + "href": "in-development/earthdata-python-r-handoff.html#dev-notes", + "title": "earthdata: Python-R Handoff", + "section": "Dev notes", + "text": "Dev notes\n\nChat with Andy May 26\nMaybe have a python script that takes arguments, compiled in a way that then in MatLab you can sys.admin that python script. Then he doesn’t need to know python\nOther approach would be MatLab to re-write earthdata in MatLab\nOur dream, revised: the code should be language-agnostic" }, { - "objectID": "tutorials/Data_stories_template.html#overview", - "href": "tutorials/Data_stories_template.html#overview", - "title": "Title", - "section": "", - "text": "Add Summary of data story/background information needed. 2-5 sentences." - }, - { - "objectID": "tutorials/Data_stories_template.html#learning-objectives", - "href": "tutorials/Data_stories_template.html#learning-objectives", - "title": "Title", - "section": "Learning Objectives", - "text": "Learning Objectives\n\nenter objective\nenter objective\n…" - }, - { - "objectID": "tutorials/Data_stories_template.html#requirements", - "href": "tutorials/Data_stories_template.html#requirements", - "title": "Title", - "section": "Requirements", - "text": "Requirements\n\n1. Compute environment - This notebook can only be run in an AWS Cloud instance running in the us-west-2 region.\n\n\n2. Earthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up. ### 3. Additional Requirements …\n\n\nInstall Packages\n\nimport earthaccess\nimport xarray as xr\n\nWe recommend authenticating your Earthdata Login (EDL) information using the earthaccess python package as follows:\n\nauth = earthaccess.login(strategy=\"netrc\") # works if the EDL login already been persisted to a netrc\nif not auth.authenticated:\n # ask for EDL credentials and persist them in a .netrc file\n auth = earthaccess.login(strategy=\"interactive\", persist=True)" + "objectID": "in-development/earthdata-python-r-handoff.html#background", + "href": "in-development/earthdata-python-r-handoff.html#background", + "title": "earthdata: Python-R Handoff", + "section": "Background", + "text": "Background\nThis was Luis’ original example code, but it downloads data. The examples above access it in the cloud. From https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/earthdata-access-demo.html\nfrom earthdata import Auth, DataGranules, Store\n\n# first we authenticate with NASA EDL\nauth = Auth().login(strategy=\"netrc\")\n\n# Then we build a Query with spatiotemporal parameters\nGranuleQuery = DataGranules().concept_id(\"C1575731655-LPDAAC_ECS\").bounding_box(-134.7,58.9,-133.9,59.2)\n\n# We get the metadata records from CMR\ngranules = GranuleQuery.get()\n\n# Now it{s time to download (or open) our data granules list with get()\nfiles = Store(auth).get(granules, local_path='./data')" }, { - "objectID": "index.html", - "href": "index.html", - "title": "NASA Earthdata Cloud Cookbook", + "objectID": "in-development/matlab.html", + "href": "in-development/matlab.html", + "title": "MATLAB Access Single NASA EarthData L2 NetCDF", "section": "", - "text": "Welcome to the NASA Openscapes Earthdata Cloud Cookbook!\nThis Cookbook is learning-oriented to support scientific researchers who use NASA Earthdata as NASA migrates data and workflows to the cloud. It has resources to learn how to work with NASA Earthdata in the cloud, as well as documentation for how to contribute to these resources. It is under active, open development." + "text": "This tutorial was written by Erin Robinson, Luis Lopez (NASA Openscapes) and Cedric Wannaz, Lisa Kempler (Mathworks)." }, { - "objectID": "index.html#welcome", - "href": "index.html#welcome", - "title": "NASA Earthdata Cloud Cookbook", - "section": "", - "text": "Welcome to the NASA Openscapes Earthdata Cloud Cookbook!\nThis Cookbook is learning-oriented to support scientific researchers who use NASA Earthdata as NASA migrates data and workflows to the cloud. It has resources to learn how to work with NASA Earthdata in the cloud, as well as documentation for how to contribute to these resources. It is under active, open development." + "objectID": "in-development/matlab.html#launch-matlab-in-2i2c-jupyterhub", + "href": "in-development/matlab.html#launch-matlab-in-2i2c-jupyterhub", + "title": "MATLAB Access Single NASA EarthData L2 NetCDF", + "section": "Launch MATLAB in 2i2c JupyterHub", + "text": "Launch MATLAB in 2i2c JupyterHub\nYou’re able to use MATLAB in the 2i2c JupyterHub; you are able to bring your own license. If you’d like to access MATLAB from the Amazon Web Services Console, see details at the bottom of this page.\n\n\n\nSelect MATLAB from the Server Options and press Start\n\n\n\n\n\nLaunch MATLAB from the JupyterHub interface by double-clicking\n\n\n\n\n\nEnter your MATLAB login information" }, { - "objectID": "index.html#the-cloud-paradigm", - "href": "index.html#the-cloud-paradigm", - "title": "NASA Earthdata Cloud Cookbook", - "section": "The Cloud Paradigm", - "text": "The Cloud Paradigm\nNASA Earthdata is available from Earth Observing System Data and Information System (EOSDIS) Distributed Active Archive Centers (DAACs) that are in the process of moving data holdings to a cloud platform. In the new paradigm, data storage (orange in the illustration) will migrate to the cloud (green) and DAAC-provided tools and services built on top of the data are co-located in the Earthdata Cloud.\n\n\n\nIllustration by Catalina Oaida, PO.DAAC\n\n\nAs this data migration occurs, DAACs will have more information about how users can access data. For example, the Cloud Data page at PO.DAAC offers access to resources to help guide data users in discovering, accessing, and utilizing cloud data. During this transition, some data will continue to be available from the traditional on premise archive, while some data will also be available from and within the Earthdata Cloud.\nLearn More About NASA’s Earthdata Cloud Migration\n\nNASA EarthData Cloud Evolution - Starting point for understanding the background for moving to the Cloud.\nHow We Work - Nice overview of EOSDIS work and emphasis on open science by Katie Baynes, Keynote at FedGeoDay2020 (video).\nCloud Data Egress: How EOSDIS Supports User Needs - Describes supporting users’ needs during cloud migration by providing analysis-ready data by Emily Cassidy (blog)." + "objectID": "in-development/matlab.html#credentials", + "href": "in-development/matlab.html#credentials", + "title": "MATLAB Access Single NASA EarthData L2 NetCDF", + "section": "Credentials", + "text": "Credentials\nDirect S3 access is achieved by passing NASA supplied temporary credentials to AWS so we can interact with S3 objects from applicable EarthData Cloud buckets. For now, each NASA DAAC has different AWS credentials endpoints. Below are some of the credential endpoints to various DAACs:\n\nPODAAC: https://archive.podaac.earthdata.nasa.gov/s3credentials\nGESDISC: https://data.gesdisc.earthdata.nasa.gov/s3credentials\nLPDAAC: https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials\nORNLDAAC: https://data.ornldaac.earthdata.nasa.gov/s3credentials\nGHRCDAAC: https://data.ghrc.earthdata.nasa.gov/s3credentials\nNSIDC: https://data.nsidc.earthdatacloud.nasa.gov/s3credentials\n\nYou will need your EarthData login to access these links. Create an account here: https://urs.earthdata.nasa.gov/users/new." }, { - "objectID": "index.html#about", - "href": "index.html#about", - "title": "NASA Earthdata Cloud Cookbook", - "section": "About", - "text": "About\nThis Earthdata Cloud Cookbook is being developed as a cross-DAAC collaboration by the NASA-Openscapes Team. Learn more at nasa-openscapes.github.io." + "objectID": "in-development/matlab.html#prerequisites", + "href": "in-development/matlab.html#prerequisites", + "title": "MATLAB Access Single NASA EarthData L2 NetCDF", + "section": "Prerequisites", + "text": "Prerequisites\n\nSave the following function in a text file called loadAWSCredentials.txt to your MATLAB working folder or to any folder that is on MATLAB search path.\n\nfunction s3Credentials = loadAWSCredentials(daacCredentialsEndpoint, login, password)\n if nargin < 2 || isempty(login)\n login = getenv('EARTHDATA_LOGIN') ;\n end\n if nargin < 3 || isempty(password)\n password = getenv('EARTHDATA_PASSWORD') ;\n end\n\n % Get S3 credentials from EarthData\n opts = weboptions('ContentType', 'json', 'HeaderFields', ...\n {'Authorization', ['Basic ',matlab.net.base64encode([login,':',password])]});\n s3Credentials = webread(daacCredentialsEndpoint, opts) ;\n\n % Set relevant environment variables with AWS credentials/region\n setenv('AWS_ACCESS_KEY_ID', s3Credentials.accessKeyId) ;\n setenv('AWS_SECRET_ACCESS_KEY', s3Credentials.secretAccessKey) ;\n setenv('AWS_SESSION_TOKEN', s3Credentials.sessionToken) ;\n setenv('AWS_DEFAULT_REGION', 'us-west-2') ;\nend\n\nDefine environment variables that specify your EarthData credentials by executing the following (updated with your credentials):\n\nsetenv(\"EARTHDATA_LOGIN\",\"your_earthdata_login\");\nsetenv(\"EARTHDATA_PASSWORD\",\"your_earthdata_password\");\nThis can be done manually in the command window, in your MATLAB startup script, or in any MATLAB function/script that is executed before calling loadAWSCredentials. We recommend not to do it from your project main script, so you can share it without divulging your credentials.\n\nSearch NASA EarthData and find the S3 link you want to access." }, { - "objectID": "index.html#citation", - "href": "index.html#citation", - "title": "NASA Earthdata Cloud Cookbook", - "section": "Citation", - "text": "Citation\nPlease cite the NASA Earthdata Cloud Cookbook through the project’s Zenodo archive using DOI: 10.5281/zenodo.7786710. This DOI represents all versions, and will always resolve to the latest one.\nThe citation will look something like:\n\nAndy Barrett, Chris Battisto, Brandon Bottomley, Aaron Friesz, Alexis Hunzinger, Mahsa Jami, Alex Lewandowski, Bri Lind, Luis López, Jack McNelis, Cassie Nickles, Catalina Oaida Taglialatela, Celia Ou, Brianna Pagán, Sargent Shriver, Amy Steiker, Michele Thornton, Makhan Virdi, Jessica Nicole Welch, Jess Welch, Erin Robinson, Julia Stewart Lowndes. (2023). NASA-Openscapes/earthdata-cloud-cookbook: NASA EarthData Cloud Cookbook v2023.03.0 (v2023.03.0). Zenodo. https://doi.org/10.5281/zenodo.7786711\n\nPlease visit the Cookbook’s DOI link to get the most recent version - the one above is not automatically generated and may be out of date as we release updated versions of the Cookbook." + "objectID": "in-development/matlab.html#accessing-data", + "href": "in-development/matlab.html#accessing-data", + "title": "MATLAB Access Single NASA EarthData L2 NetCDF", + "section": "Accessing data", + "text": "Accessing data\nMATLAB HDF5 interfaces can access netCDF4 files stored in S3/Azure, while the netCDF interfaces cannot. The example below shows how to access a netCDF file using the HDF5 interfaces. SEe also this video: MATLAB and HDF5: Compression, Cloud, and Community – Ellen Johnson, MathWorks.\nMATLAB provides two interfaces for accessing HDF5 datasets:\n\nHigh-level access functions make it easy to read and view data in an HDF5 file or write a variable from the MATLAB® workspace into an HDF5 file\nLow-level functions in the HDF5 library packages provide direct access to the more than 300 functions in the HDF5 C library from the HDF Group.\n\nLow-level functions allow you to access advanced HDF5 functionality and bypasses I/O overhead in the high-level interface, but has a substantially higher learning curve unless already familiar with the HDF5 C API.\n\nExample\n\nFind the relevant DAAC/endpoint for the dataset that you want to access. For the following dataset:\n\ns3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20100619062008-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\nThe relevant DAAC is PODAAC and the relevant endpoint is:\nhttps://archive.podaac.earthdata.nasa.gov/s3credentials\n\nSet AWS credentials using the loadAWSCredentials function, passing the relevant endpoint:\n\ndaacCredentialsEndpoint = \"https://archive.podaac.earthdata.nasa.gov/s3credentials\";\nloadAWSCredentials(daacCredentialsEndpoint);\nThese credentials are temporary; they expire every 30 minutes or so and must be refreshed periodically.\n\nAccess data Define path of relevant file and dataset:\n\nFILE_NAME = \"s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20100619062008-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\";\nDATASET_NAME = \"sea_surface_temperature_4um\";\nDATASET_PATH = \"/\" + DATASET_NAME ;\nFirst, check that you can access the dataset:\nh5disp(FILE_PATH);\nIf credentials and paths are correct, h5disp will display the structure of the data file.\n\n\nUsing MATLAB high-level HDF5 interface\nunits = h5readatt(FILE_NAME,DATASET_PATH,\"units\");\nfill_value = h5readatt(FILE_NAME,DATASET_PATH,\"_FillValue\");\nname = h5readatt(FILE_NAME,DATASET_PATH,\"long_name\");\ndata = h5read(FILE_NAME,DATASET_PATH);\ndata(data == fill_value) = NaN;\nlat = h5read(FILE_NAME,\"/lat\");\nlon = h5read(FILE_NAME,\"/lon\");\nBuild a map using contour:\ncontour(lon,lat,data);\ntitle(sprintf(\"%s [%s]\",name,units));\n\n\n\nMATLAB example map\n\n\n\n\nUsing MATLAB low-level HDF5 interface\n% Open file and root group\nfile_id = H5F.open(FILE_NAME,\"H5F_ACC_RDONLY\",\"H5P_DEFAULT\");\nroot_id = H5G.open(file_id,\"/\") ;\n\n% Open/read sea surface temperature (SST) dataset\ndata_id = H5D.open(file_id,DATASET_PATH);\ndata=H5D.read(data_id,\"H5T_NATIVE_DOUBLE\",\"H5S_ALL\",\"H5S_ALL\",\"H5P_DEFAULT\");\n\n% Open/read/close SST units attribute\nattr_id = H5A.open_by_name(root_id,DATASET_NAME,\"units\");\nunits = H5A.read(attr_id,\"H5ML_DEFAULT\");\nH5A.close(attr_id);\n\n% Open/read/close SST long_name attribute\nattr_id = H5A.open_by_name(root_id,DATASET_NAME,\"long_name\");\nlong_name = H5A.read(attr_id,\"H5ML_DEFAULT\");\nH5A.close(attr_id);\n\n% Open/read/close SST _FillValue attribute\nattr_id = H5A.open_by_name(root_id,DATASET_NAME,\"_FillValue\");\nfillvalue = H5A.read(attr_id,\"H5T_NATIVE_DOUBLE\");\nH5A.close(attr_id);\n\n% Close SST dataset and root group\nH5D.close(data_id);\nH5G.close(root_id);\n\n% Open/read/close latitude dataset\nlat_id = H5D.open(file_id,\"lat\");\nlat = H5D.read(lat_id,\"H5T_NATIVE_DOUBLE\",\"H5S_ALL\",\"H5S_ALL\",\"H5P_DEFAULT\");\nH5D.close(lat_id);\n\n% Open/read/close longitude dataset\nlon_id = H5D.open(file_id,\"lon\");\nlon = H5D.read(lon_id,\"H5T_NATIVE_DOUBLE\",\"H5S_ALL\",\"H5S_ALL\",\"H5P_DEFAULT\");\nH5D.close(lon_id);\n\n% Close file\nH5F.close(file_id);\n\n% Replace fill value with NaN\ndata(data==fillvalue) = NaN;\n\n% Plot contour map\ncontour(lon,lat,data);\ntitle(sprintf(\"%s [%s]\",long_name{1},units{1}));" }, { - "objectID": "index.html#license", - "href": "index.html#license", - "title": "NASA Earthdata Cloud Cookbook", - "section": "License", - "text": "License\nOpenscapes makes all of their materials publicly available under open source licenses.\nOur License is adapted from The Turing Way" + "objectID": "in-development/matlab.html#launch-with-aws-console", + "href": "in-development/matlab.html#launch-with-aws-console", + "title": "MATLAB Access Single NASA EarthData L2 NetCDF", + "section": "Launch with AWS Console", + "text": "Launch with AWS Console\nAn alternative is to launch with the AWS Console. This chapter is for research teams currently working in MATLAB with NASA EarthData and wanting to take advantage of doing analysis in the Cloud. Our initial focus is on Amazon Web Services (AWS). For general, background on the Cloud and NASA Earthdata’s migration to the Cloud, checkout earlier chapters of the cookbook [TO DO: Add a link and specific reference].\n\nPrerequisites\n\nMATLAB License - You must have access to a MATLAB license that allows for access to the cloud. To check your license, in the MATLAB command line enter:\nver -support\nAWS Account - For the current experiments, we will access the AWS console, so you need an AWS account. It is free to set up, but will require a credit card linked to the account. https://aws.amazon.com/\nAWS Region: We are working in AWS US-West-2 because NASA Earthdata is hosted in US-West-2. If there is an option to pick an AWS region, pick US-West-2.\nGitHub Account - If you want to use GitHub as part of your workflow and don’t already have an account, create a GitHub account: https://github.com/\nNASA Earthdata login - Create an account here: https://urs.earthdata.nasa.gov/users/new\nMicrosoft Remote Desktop - When we spin up the AWS virtual machine with MATLAB, we will need to have a remote desktop option. These how-to’s were done on a mac and we tested the free Microsoft Remote Desktop. Download for macs\n\n\n\nPrerequisite: AWS EC2 Key Pair\nThis isn’t included in the numbered prereq list above because it is a bit more involved than creating an account or knowing what region we work in. You need to create an SSH Key Pair in the region you want to work in. This is a one-time step that you won’t need to do each time you launch the stack.\n\nLog in to AWS\nIn the search box enter, ‘Create AWS Key Pair.’ Under features choose ‘Key Pairs’ (AWS help)\n\nCheck that the top bar says, ‘Oregon’ (AKA - US-West-2) and if not, click that down arrow to choose US-West-2. Then click the orange ‘Create key pair’ button.\n\nTroubleshooting tip: If you create the key outside of the region you want to work in, it will not show up when you launch the stack below.\nOn the form - give your key a name, choose the RSA and PEM options (these should be the defaults.)\n\nOk - now we are ready to start!\n\n\nCreating the AWS Stack with MATLAB\nNote: The first time I launched the AWS Stack it seemed to take for-ever. I thought “how could I possibly need to do this every time I want to use MATLAB in the cloud?” It does speed up, eventually it get’s a bit faster. I also have learned to plan a bit better - if I know I want to do some work, I get the launched, have coffee, and when I come back it’s ready to roll.\n\nFrom this MATLAB GitHub page click the release for 2022a under deployment steps.\nThis brings up a list of MATLAB on Amazon Web Services (Linux VM). Choose & click the ‘launch stack’ link for US-West-2.\n\nThis opens the ‘Quick create stack’ form based on the MATLAB template. That means that when you launch this stack it will come with MATLAB on the desktop. Fill out the form to create the AWS stack:\n\nGive the stack a name like ‘matlab-test’\nKeep the pre-filled options the same for now.\nRemote Access:\n\n“Allow Connections From:” You will need to know your IP address. You can google, “what’s my IP address?”\nEnter your IP address followed by a /32 like this -> [my.IP.address/32]\nIn the SSH Key Pair - the key pair you created above should show up in the drop down. If it doesn’t show up, see the troubleshooting tip.\nPick a remote password. This is not your AWS password or your GitHub password, this is the password that you will use to login with the microsoft remote desktop (username: ubuntu)\n\nNetwork configuration\n\nThere is one VPC option - choose that\nFor subnet - I pick the first one and it works. So pick the first option.\n\nAutoshutdown hasn’t worked for me so far, so for now I leave this set as never and delete the stack when I am finished.\nCheck the box that “I acknowledge that AWS CloudFormation might create IAM resources.”\nClick ‘Create stack’\nWait…. [~ 10 minutes]\n\nYou can check the status by clicking the refresh button on the right corner\n\n\n\n\nLaunch the AWS Stack with Microsoft Remote Desktop\n\nOnce the stack is created it will say ‘Create_complete’ on the left side.\n\nClick the outputs tab and copy the value text. It will start with ‘ec2-…’\nOpen Microsoft Remote Desktop\n\nClick the + to add a PC\nPaste the value text as the PC Name\nClick on the grey box of your new computer in the remote desktop window\nA login will pop up\n\n\nUsername is ubuntu\nPassword is the password you set in 3.3.3 above in the section on ‘Creating AWS Stack’\n\nA certificate message will pop up - say ok\nThe desktop will launch\n\nWait … [~2 mins]\n\n\n\nOpen MATLAB on Remote Desktop\n\nClick the MATLAB icon on the remote desktop\n\nWait … [~4 mins]\nLogin with your MATLAB credentials\nYou are in!\n\n\n\nShutdown your AWS Stack\nAfter each session you need to turn off the AWS Stack. If you forget this step and leave it running it is like keeping a computer on for the month. For the large instance it costs $0.5/day so it’s a few dollars a month.\n\nGo back to AWS\nSearch for stack\nClick on the name of your stack\nClick ‘Delete’\n\nConfirm the delete" }, { - "objectID": "in-development/CMR-STAC-Search.html", - "href": "in-development/CMR-STAC-Search.html", - "title": "Getting Started with NASA’s CMR-STAC API in R", + "objectID": "environment-setup/index.html", + "href": "environment-setup/index.html", + "title": "Cloud Environment Setup", "section": "", - "text": "This tutorial demonstrates how to interact with CMR-STAC in R.\nThis tutorial will teach you how to navigate and explore NASA’s Common Metadata Repository (CMR) SpatioTemporal Asset Catalog (STAC) to learn about the datasets available through LP DAAC Cumulus cloud archive." - }, - { - "objectID": "in-development/CMR-STAC-Search.html#a.-what-is-stac", - "href": "in-development/CMR-STAC-Search.html#a.-what-is-stac", - "title": "Getting Started with NASA’s CMR-STAC API in R", - "section": "1a. What is STAC?", - "text": "1a. What is STAC?\nSTAC is short for Spatiotemporal Asset Catalog, a series of specifications that provide a common language for interpreting geospatial information in order to standardize indexing and discovery of spatiotemporal assets (files containing information about the Earth across space and time).\nThere are four specifications that work both independently and together:\n\nSTAC Catalog\nSTAC Collection\nSTAC Item\n\nSTAC API specification builds on top of the three core specifications mentioned above. All these specifications are intended to be used together, yet are designed in a way that each piece is small, self-contained, and reusable in other contexts." - }, - { - "objectID": "in-development/CMR-STAC-Search.html#b.-what-is-the-cmr-stac-api", - "href": "in-development/CMR-STAC-Search.html#b.-what-is-the-cmr-stac-api", - "title": "Getting Started with NASA’s CMR-STAC API in R", - "section": "1b. What is the CMR-STAC API?", - "text": "1b. What is the CMR-STAC API?\nThe Common Metadata Repository (CMR) is a metadata system that catalogs Earth Science data and associated metadata records. NASA’s CMR-STAC Application Programming Interface (API) is a translation API for STAC users who want to access and search through CMR’s vast metadata holdings using STAC keywords." - }, - { - "objectID": "in-development/CMR-STAC-Search.html#a.-cmr-stac-api", - "href": "in-development/CMR-STAC-Search.html#a.-cmr-stac-api", - "title": "Getting Started with NASA’s CMR-STAC API in R", - "section": "2a. CMR-STAC API", - "text": "2a. CMR-STAC API\nThe CMR-STAC API contains endpoints that enable the querying of STAC items.\nAssign the CMR-STAC URL to a static variable.\n\nCMR_STAC_URL <- 'https://cmr.earthdata.nasa.gov/stac/'\n\nConnect to the CMR-STAC landing page which contains all the available data providers and their STAC endpoint. In this tutorial, the httr package is used to navigate CMR-STAC API.\n\ncmr_cat <- httr::GET(CMR_STAC_URL) %>% # Request and retrieve the info from CMR-STAC URL\n httr::content() \ncat('You are using',cmr_cat$title,'version',cmr_cat$stac_version,\".\", cmr_cat$description,sep=\" \")\n\nYou are using NASA CMR STAC Proxy version 1.0.0 . This is the landing page for CMR-STAC. Each provider link contains a STAC endpoint.\n\n\nHere, jsonlite is used to change the format of the content returned from our request and the DT package is used to make the returned information more readable. The providers’ names and URL links are found in the title and ‘href’ fields respectively.\n\ncmr_cat_links <- cmr_cat$links %>% \n jsonlite::toJSON(auto_unbox = TRUE) %>% \n jsonlite::fromJSON() %>% \n as.data.frame()\nDT::datatable(cmr_cat_links)\n\n\n\n\n\n\nThe data frame above shows all the data providers with their associated STAC catalog endpoints. You will notice above that the CMR-STAC API contains many different endpoints–not just from NASA LP DAAC, but also contains endpoints for other NASA ESDIS DAACs. Use the title field to identify the data provider you are interested in. The data product used in this tutorial is hosted in the LP DAAC Cumulus Cloud space (LPCLOUD).\nAssign LPCLOUD to the provider variable and get this provider’s endpoint from the CMR catalog using the URL in Link field.\n\nprovider <- 'LPCLOUD'\nlpcloud_cat_link <- cmr_cat_links[which(cmr_cat_links$title == provider), 'href']\nlpcloud_cat_link\n\n[1] \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD\"" - }, - { - "objectID": "in-development/CMR-STAC-Search.html#b.-stac-catalog", - "href": "in-development/CMR-STAC-Search.html#b.-stac-catalog", - "title": "Getting Started with NASA’s CMR-STAC API in R", - "section": "2b. STAC Catalog", - "text": "2b. STAC Catalog\nSTAC Catalog Contains a JSON file of links that organize all the available collections. Below, connect to the LPCLOUD STAC Catalog endpoint using httr package and print the information contained in the Catalog.\n\nlpcloud_cat <- httr::GET(lpcloud_cat_link) %>% \n httr::content()\n\nlpcloud_cat <- lpcloud_cat %>% \n jsonlite::toJSON(auto_unbox = TRUE) %>% \n jsonlite::fromJSON() \n\nDT::datatable(lpcloud_cat$links)\n\n\n\n\n\n\nLPCLOUD STAC catalog includes URL links to the root, collections, search, and child STAC Catalogs. The data frame above also shows the available collections in the LPCLOUD catalog." - }, - { - "objectID": "in-development/CMR-STAC-Search.html#c.-stac-collection", - "href": "in-development/CMR-STAC-Search.html#c.-stac-collection", - "title": "Getting Started with NASA’s CMR-STAC API in R", - "section": "2c. STAC Collection", - "text": "2c. STAC Collection\nSTAC Collection is extension of STAC Catalog containing additional information that describe the STAC Items in that Collection.\nGet the URL link to the STAC Collections.\n\nlpcloud_col_link <- lpcloud_cat$links[which(lpcloud_cat$links$rel == 'collections'),'href']\nlpcloud_col_link\n\n[1] \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections\"\n\n\nNext, get the content describing the collections within LPCLOUD Catalog. Important information such as data collection ID, title, description, and links to collection endpoints are provided here.\n\nlpcloud_collection <- httr::GET(lpcloud_col_link) %>% \n httr::content() \n\nlpcloud_collection <- lpcloud_collection %>% \n jsonlite::toJSON(auto_unbox = TRUE, pretty = TRUE)\n\nPrint the collections within LPCLOUD STAC catalog.\n\nlpcloud_collection_df <- jsonlite::fromJSON(lpcloud_collection)$collections\nlpcloud_collection_df$id\n\n [1] \"ASTGTM.v003\" \"ECO_L1B_ATT.v002\" \"ECO_L2_CLOUD.v002\"\n [4] \"ECO_L1B_GEO.v002\" \"ECO_L2_LSTE.v002\" \"ECO_L1B_RAD.v002\" \n [7] \"ECO_L2T_LSTE.v002\" \"EMITL1BRAD.v001\" \"EMITL2ARFL.v001\" \n[10] \"HLSL30.v2.0\" \n\n\nIn CMR, Collection ID is used to query by a specific product, so be sure to save the ID for a collection you are interested in. For instance, the Collection ID for ASTER Global Digital Elevation Model V003 is ASTGTM.v003. Note that the “id” shortname is in the format: productshortname.vVVV (where VVV = product version).\nHere, get the URL link to the ASTGTM.v003 STAC Collection. If you are interested in querying a different LPCLOUD product, swap out the shortname to assign to the collection variable below.\n\ncollection <- 'ASTGTM.v003' # USER INPUT\ncol_links <- lpcloud_collection_df$links[which(lpcloud_collection_df$id == collection)] %>% \n as.data.frame()\n\nastgtm_URL <- col_links[which(col_links$rel == 'self'), 'href']\nastgtm_URL\n\n[1] \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/ASTGTM.v003\"\n\n\nThe STAC Collection metadata for any collection contains metadata and information that is applicable to every STAC Item and asset(s) that it contains. Get the content of the ASTGTM.v003 collection URL and print the collection description.\n\nastgtm_collection <- httr::GET(astgtm_URL) %>% \n httr::content()\n\nastgtm_collection <- astgtm_collection %>% \n jsonlite::toJSON(auto_unbox = TRUE) %>% \n jsonlite::fromJSON()\n\ncat(astgtm_collection$description)\n\nThe ASTER Global Digital Elevation Model (GDEM) Version 3 (ASTGTM) provides a global digital elevation model (DEM) of land areas on Earth at a spatial resolution of 1 arc second (approximately 30 meter horizontal posting at the equator).\n\nThe development of the ASTER GDEM data products is a collaborative effort between National Aeronautics and Space Administration (NASA) and Japan’s Ministry of Economy, Trade, and Industry (METI). The ASTER GDEM data products are created by the Sensor Information Laboratory Corporation (SILC) in Tokyo. \n\nThe ASTER GDEM Version 3 data product was created from the automated processing of the entire ASTER Level 1A (https://doi.org/10.5067/ASTER/AST_L1A.003) archive of scenes acquired between March 1, 2000, and November 30, 2013. Stereo correlation was used to produce over one million individual scene based ASTER DEMs, to which cloud masking was applied. All cloud screened DEMs and non-cloud screened DEMs were stacked. Residual bad values and outliers were removed. In areas with limited data stacking, several existing reference DEMs were used to supplement ASTER data to correct for residual anomalies. Selected data were averaged to create final pixel values before partitioning the data into 1 degree latitude by 1 degree longitude tiles with a one pixel overlap. To correct elevation values of water body surfaces, the ASTER Global Water Bodies Database (ASTWBD) (https://doi.org/10.5067/ASTER/ASTWBD.001) Version 1 data product was also generated. \n\nThe geographic coverage of the ASTER GDEM extends from 83° North to 83° South. Each tile is distributed in GeoTIFF format and projected on the 1984 World Geodetic System (WGS84)/1996 Earth Gravitational Model (EGM96) geoid. Each of the 22,912 tiles in the collection contain at least 0.01% land area. \n\nProvided in the ASTER GDEM product are layers for DEM and number of scenes (NUM). The NUM layer indicates the number of scenes that were processed for each pixel and the source of the data.\n\nWhile the ASTER GDEM Version 3 data products offer substantial improvements over Version 2, users are advised that the products still may contain anomalies and artifacts that will reduce its usability for certain applications. \n\nImprovements/Changes from Previous Versions \n• Expansion of acquisition coverage to increase the amount of cloud-free input scenes from about 1.5 million in Version 2 to about 1.88 million scenes in Version 3.\n• Separation of rivers from lakes in the water body processing. \n• Minimum water body detection size decreased from 1 km2 to 0.2 km2. \n\n\nWe can also get the spatial and temporal extent information. Below, we can see this collection has a global spatial extent. ASTER GDEM is a single, static dataset that incorporates observation from March 2000 to November 2013.\n\nastgtm_collection$extent %>% \n jsonlite::toJSON(auto_unbox = TRUE)\n\n{\"spatial\":{\"bbox\":[[-180,-83,180,82]]},\"temporal\":{\"interval\":[[\"2000-03-01T00:00:00.000Z\",\"2013-11-30T23:59:59.999Z\"]]}} \n\n\nSTAC collection also includes useful links. You can visit all the items within this collection using the Items URL.\n\nDT::datatable(astgtm_collection$links)\n\n\n\n\n\n\nGet the URL to the ASTGTM.v003 Items.\n\nitems_url <- astgtm_collection$links [which(astgtm_collection$links$rel == 'items'), 'href']\nitems_url\n\n[1] \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/ASTGTM.v003/items\"" - }, - { - "objectID": "in-development/CMR-STAC-Search.html#d.-stac-item", - "href": "in-development/CMR-STAC-Search.html#d.-stac-item", - "title": "Getting Started with NASA’s CMR-STAC API in R", - "section": "2d. STAC Item", - "text": "2d. STAC Item\nSTAC Item represents data and metadata assets that are spatiotemporally coincident. Below, query the STAC Items within the ASTGTM.v003 STAC Collection and print the first item in the collection.\n\nastgtm_items <- httr::GET(items_url) %>% \n httr::content(as = \"text\") %>% \n jsonlite::fromJSON()\n\nF1 <- astgtm_items$features[1,] %>% \n jsonlite::toJSON(auto_unbox = TRUE, pretty = TRUE)\nF1\n\n[\n {\n \"type\": \"Feature\",\n \"id\": \"ASTGTMV003_N03E008\",\n \"stac_version\": \"1.0.0\",\n \"stac_extensions\": [],\n \"collection\": \"ASTGTM.v003\",\n \"geometry\": {\n \"type\": \"Polygon\",\n \"coordinates\": [\n [\n [7.9999, 2.9999],\n [9.0001, 2.9999],\n [9.0001, 4.0001],\n [7.9999, 4.0001],\n [7.9999, 2.9999]\n ]\n ]\n },\n \"bbox\": [7.9999, 2.9999, 9.0001, 4.0001],\n \"links\": [\n {\n \"rel\": \"self\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/ASTGTM.v003/items/ASTGTMV003_N03E008\"\n },\n {\n \"rel\": \"parent\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/ASTGTM.v003\"\n },\n {\n \"rel\": \"collection\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/ASTGTM.v003\"\n },\n {\n \"rel\": \"root\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/\"\n },\n {\n \"rel\": \"provider\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD\"\n },\n {\n \"rel\": \"via\",\n \"href\": \"https://cmr.earthdata.nasa.gov/search/concepts/G1716133754-LPCLOUD.json\"\n },\n {\n \"rel\": \"via\",\n \"href\": \"https://cmr.earthdata.nasa.gov/search/concepts/G1716133754-LPCLOUD.umm_json\"\n }\n ],\n \"properties\": {\n \"datetime\": \"2000-03-01T00:00:00.000Z\",\n \"start_datetime\": \"2000-03-01T00:00:00.000Z\",\n \"end_datetime\": \"2013-11-30T23:59:59.000Z\"\n },\n \"assets\": {\n \"003/ASTGTMV003_N03E008_dem\": {\n \"href\": \"https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/ASTGTM.003/ASTGTMV003_N03E008_dem.tif\",\n \"title\": \"Download ASTGTMV003_N03E008_dem.tif\"\n },\n \"003/ASTGTMV003_N03E008_num\": {\n \"href\": \"https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/ASTGTM.003/ASTGTMV003_N03E008_num.tif\",\n \"title\": \"Download ASTGTMV003_N03E008_num.tif\"\n },\n \"browse\": {\n \"href\": \"https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/ASTGTM.003/ASTGTMV003_N03E008.1.jpg\",\n \"type\": \"image/jpeg\",\n \"title\": \"Download ASTGTMV003_N03E008.1.jpg\"\n },\n \"metadata\": {\n \"href\": \"https://cmr.earthdata.nasa.gov/search/concepts/G1716133754-LPCLOUD.xml\",\n \"type\": \"application/xml\"\n },\n \"003/ASTGTMV003_N02E022_dem\": {},\n \"003/ASTGTMV003_N02E022_num\": {},\n \"003/ASTGTMV003_N00W065_dem\": {},\n \"003/ASTGTMV003_N00W065_num\": {},\n \"003/ASTGTMV003_N01E009_dem\": {},\n \"003/ASTGTMV003_N01E009_num\": {},\n \"003/ASTGTMV003_N02E009_dem\": {},\n \"003/ASTGTMV003_N02E009_num\": {},\n \"003/ASTGTMV003_N03E021_dem\": {},\n \"003/ASTGTMV003_N03E021_num\": {},\n \"003/ASTGTMV003_N01E021_dem\": {},\n \"003/ASTGTMV003_N01E021_num\": {},\n \"003/ASTGTMV003_N01E042_dem\": {},\n \"003/ASTGTMV003_N01E042_num\": {},\n \"003/ASTGTMV003_N01W069_dem\": {},\n \"003/ASTGTMV003_N01W069_num\": {},\n \"003/ASTGTMV003_N01W080_dem\": {},\n \"003/ASTGTMV003_N01W080_num\": {}\n }\n }\n] \n\n\nNotice that the number of items matching our request is far more than what is returned.\n\ncat(astgtm_items$context$matched, 'items matched your request but', astgtm_items$context$returned, 'items are returned.')\n\n22912 items matched your request but 10 items are returned.\n\n\nThis is because the return is paginated. The STAC API, by default, returns the first 10 records. To explore more items, you can add ?page=n (in which n is the page number (i.e. ?page=2)) to the URL link and submit another request. Below, request a query to return records on the second page.\n\npage_2_url <- paste0(items_url, '?page=2')\n\nastgtm_page2_items <- httr::GET(page_2_url) %>% \n httr::content(as = \"text\") %>% \n jsonlite::fromJSON()\n\nastgtm_page2_items$features[1,] %>% \n jsonlite::toJSON(auto_unbox = TRUE, pretty = TRUE)\n\n[\n {\n \"type\": \"Feature\",\n \"id\": \"ASTGTMV003_N03E042\",\n \"stac_version\": \"1.0.0\",\n \"stac_extensions\": [],\n \"collection\": \"ASTGTM.v003\",\n \"geometry\": {\n \"type\": \"Polygon\",\n \"coordinates\": [\n [\n [41.9999, 2.9999],\n [43.0001, 2.9999],\n [43.0001, 4.0001],\n [41.9999, 4.0001],\n [41.9999, 2.9999]\n ]\n ]\n },\n \"bbox\": [41.9999, 2.9999, 43.0001, 4.0001],\n \"links\": [\n {\n \"rel\": \"self\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/ASTGTM.v003/items/ASTGTMV003_N03E042\"\n },\n {\n \"rel\": \"parent\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/ASTGTM.v003\"\n },\n {\n \"rel\": \"collection\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/ASTGTM.v003\"\n },\n {\n \"rel\": \"root\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/\"\n },\n {\n \"rel\": \"provider\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD\"\n },\n {\n \"rel\": \"via\",\n \"href\": \"https://cmr.earthdata.nasa.gov/search/concepts/G1726373735-LPCLOUD.json\"\n },\n {\n \"rel\": \"via\",\n \"href\": \"https://cmr.earthdata.nasa.gov/search/concepts/G1726373735-LPCLOUD.umm_json\"\n }\n ],\n \"properties\": {\n \"datetime\": \"2000-03-01T00:00:00.000Z\",\n \"start_datetime\": \"2000-03-01T00:00:00.000Z\",\n \"end_datetime\": \"2013-11-30T23:59:59.000Z\"\n },\n \"assets\": {\n \"003/ASTGTMV003_N03E042_dem\": {\n \"href\": \"https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/ASTGTM.003/ASTGTMV003_N03E042_dem.tif\",\n \"title\": \"Download ASTGTMV003_N03E042_dem.tif\"\n },\n \"003/ASTGTMV003_N03E042_num\": {\n \"href\": \"https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/ASTGTM.003/ASTGTMV003_N03E042_num.tif\",\n \"title\": \"Download ASTGTMV003_N03E042_num.tif\"\n },\n \"browse\": {\n \"href\": \"https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/ASTGTM.003/ASTGTMV003_N03E042.1.jpg\",\n \"type\": \"image/jpeg\",\n \"title\": \"Download ASTGTMV003_N03E042.1.jpg\"\n },\n \"metadata\": {\n \"href\": \"https://cmr.earthdata.nasa.gov/search/concepts/G1726373735-LPCLOUD.xml\",\n \"type\": \"application/xml\"\n },\n \"003/ASTGTMV003_N00W061_dem\": {},\n \"003/ASTGTMV003_N00W061_num\": {},\n \"003/ASTGTMV003_N02W066_dem\": {},\n \"003/ASTGTMV003_N02W066_num\": {},\n \"003/ASTGTMV003_N02W069_dem\": {},\n \"003/ASTGTMV003_N02W069_num\": {},\n \"003/ASTGTMV003_N01E022_dem\": {},\n \"003/ASTGTMV003_N01E022_num\": {},\n \"003/ASTGTMV003_N01E026_dem\": {},\n \"003/ASTGTMV003_N01E026_num\": {},\n \"003/ASTGTMV003_N02W064_dem\": {},\n \"003/ASTGTMV003_N02W064_num\": {},\n \"003/ASTGTMV003_N01W064_dem\": {},\n \"003/ASTGTMV003_N01W064_num\": {},\n \"003/ASTGTMV003_N01E027_dem\": {},\n \"003/ASTGTMV003_N01E027_num\": {},\n \"003/ASTGTMV003_N00E006_dem\": {},\n \"003/ASTGTMV003_N00E006_num\": {}\n }\n }\n]" + "text": "This is primarily for Cloud environments, not locally." }, { - "objectID": "in-development/CMR-STAC-Search.html#e.-assets", - "href": "in-development/CMR-STAC-Search.html#e.-assets", - "title": "Getting Started with NASA’s CMR-STAC API in R", - "section": "2e. Assets", - "text": "2e. Assets\nThe STAC Item ID (CMR Granule ID) is the unique identifier assigned to each granule within a data collection. Within each STAC Item are assets, which include the downloadable and streamable URL to data files along with other asset objects. Below, the first Granule ID is used to get the downloadable data file.\n\nitems_df <- jsonlite::fromJSON(F1) \nitem <- items_df$assets # Get the assets for the first Item\nassets <- purrr::map_df(items_df$assets, data.frame, .id = 'asset')\nassets\n\n asset\n1 003/ASTGTMV003_N03E008_dem\n2 003/ASTGTMV003_N03E008_num\n3 browse\n4 metadata\n5 003/ASTGTMV003_N02E022_dem\n6 003/ASTGTMV003_N02E022_num\n7 003/ASTGTMV003_N00W065_dem\n8 003/ASTGTMV003_N00W065_num\n9 003/ASTGTMV003_N01E009_dem\n10 003/ASTGTMV003_N01E009_num\n11 003/ASTGTMV003_N02E009_dem\n12 003/ASTGTMV003_N02E009_num\n13 003/ASTGTMV003_N03E021_dem\n14 003/ASTGTMV003_N03E021_num\n15 003/ASTGTMV003_N01E021_dem\n16 003/ASTGTMV003_N01E021_num\n17 003/ASTGTMV003_N01E042_dem\n18 003/ASTGTMV003_N01E042_num\n19 003/ASTGTMV003_N01W069_dem\n20 003/ASTGTMV003_N01W069_num\n21 003/ASTGTMV003_N01W080_dem\n22 003/ASTGTMV003_N01W080_num\n href\n1 https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/ASTGTM.003/ASTGTMV003_N03E008_dem.tif\n2 https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/ASTGTM.003/ASTGTMV003_N03E008_num.tif\n3 https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/ASTGTM.003/ASTGTMV003_N03E008.1.jpg\n4 https://cmr.earthdata.nasa.gov/search/concepts/G1716133754-LPCLOUD.xml\n5 <NA>\n6 <NA>\n7 <NA>\n8 <NA>\n9 <NA>\n10 <NA>\n11 <NA>\n12 <NA>\n13 <NA>\n14 <NA>\n15 <NA>\n16 <NA>\n17 <NA>\n18 <NA>\n19 <NA>\n20 <NA>\n21 <NA>\n22 <NA>\n title type\n1 Download ASTGTMV003_N03E008_dem.tif <NA>\n2 Download ASTGTMV003_N03E008_num.tif <NA>\n3 Download ASTGTMV003_N03E008.1.jpg image/jpeg\n4 <NA> application/xml\n5 <NA> <NA>\n6 <NA> <NA>\n7 <NA> <NA>\n8 <NA> <NA>\n9 <NA> <NA>\n10 <NA> <NA>\n11 <NA> <NA>\n12 <NA> <NA>\n13 <NA> <NA>\n14 <NA> <NA>\n15 <NA> <NA>\n16 <NA> <NA>\n17 <NA> <NA>\n18 <NA> <NA>\n19 <NA> <NA>\n20 <NA> <NA>\n21 <NA> <NA>\n22 <NA> <NA>\n\n\nThe links found in the href field can be used to download each specific asset." + "objectID": "environment-setup/index.html#how-environments-work", + "href": "environment-setup/index.html#how-environments-work", + "title": "Cloud Environment Setup", + "section": "How environments work", + "text": "How environments work\nComing soon." }, { - "objectID": "in-development/CMR-STAC-Search.html#a.-define-search-parameters", - "href": "in-development/CMR-STAC-Search.html#a.-define-search-parameters", - "title": "Getting Started with NASA’s CMR-STAC API in R", - "section": "3a. Define Search Parameters", - "text": "3a. Define Search Parameters\nNext, define the search parameters. - Query by collection: Collection IDs should be defined as a list. - Spatial Querying via Bounding Box: A bounding box including the coordinates of LL (lower left) and UR (upper right) respectively. - Temporal Querying: Time period of interest should be specified as YYYY-MM-DDTHH:MM:SSZ/YYYY-MM-DDTHH:MM:SSZ.\n\ncollections <- list('ASTGTM.v003')\ndatetime <- '2000-01-01T00:00:00Z/2001-01-31T23:59:59Z' #YYYY-MM-DDTHH:MM:SSZ/YYYY-MM-DDTHH:MM:SSZ\nbbox <- '-122.0622682571411,39.897234301806,-122.04918980598451,39.91309383703065' # LL and UR Coordinates\n\nCreate search body object from our search parameters.\n\nbody <- list(limit=100,\n datetime=datetime,\n bbox= bbox,\n collections= collections)\n\nNotice the limit parameter in the body object. This parameter allows us to adjust the number of records returned during a request (default = 10)." + "objectID": "environment-setup/index.html#corn", + "href": "environment-setup/index.html#corn", + "title": "Cloud Environment Setup", + "section": "Corn", + "text": "Corn\nCorn is a base image that allows the provisioning of a multi-kernel Docker base image for JupyterHub deployments. corn uses the amazing Pangeo’s base image, installs all the environments it finds under ci/environments and makes them available as kernels in the base image so users can select which kernel to use depending on their needs. We’re able to update this environment leveraging GitHub Actions and deployment. Corn has streamlined a lot of environment settings and has been a big leap forward for our work in the Cloud — and you can use it too.\nCorn full information coming soon. In the meantime, see:\n\nThe why, what, and how of our NASA Openscapes cloud infrastructure: 2i2c JupyterHub and corn environment - an awesome blog post by Corn creator Luis Lopez, November 2022\nNASA Openscapes Cloud Infrastructure - awesome slides and (video) by Luis Lopez in October 2022\nCorn GitHub Repository README and Issues\n\n\n\n\nIntegration between Openscapes and 2i2c. We update the environment via GitHub Actions and Docker deployment." }, { - "objectID": "in-development/CMR-STAC-Search.html#b.-search-for-items", - "href": "in-development/CMR-STAC-Search.html#b.-search-for-items", - "title": "Getting Started with NASA’s CMR-STAC API in R", - "section": "3b. Search for Items", - "text": "3b. Search for Items\nNext, submit a query to STAC Search endpoint using a POST request.\n\nsearch_req <- httr::POST(lpcloud_search_URL[1], body = body, encode = \"json\") %>% \n httr::content(as = \"text\") %>% \n jsonlite::fromJSON()\n\nnames(search_req)\n\n[1] \"type\" \"stac_version\" \"numberMatched\" \"numberReturned\"\n[5] \"features\" \"links\" \"context\" \n\n\nLet’s see how many STAC Items, or granules, intersect with our search parameters.\n\ncat(\"The number of STAC Items matched your query is \", search_req$numberMatched, 'and ', search_req$numberReturned, 'Items are returned.')\n\nThe number of STAC Items matched your query is 1 and 1 Items are returned.\n\n\nNext, create a data frame with the returned information, including granule ID, datetime properties, and the downloadable URL links to the assets.\n\ngranule_list <- list()\n\nn <- 1\nfor(row in row.names(search_req$features)){\n f <- search_req$features[row,]\n for (b in f$assets){\n df <- data.frame(Collection = f$collection,\n Granule_ID = f$id,\n Datetime = f$properties$datetime,\n Asset_Link = b$href, stringsAsFactors=FALSE)\n granule_list[[n]] <- df\n n <- n + 1\n }\n}\n\nsearch_df <- do.call(rbind, granule_list)\nDT::datatable(search_df)\n\n\n\n\n\n\nThe CMR-STAC Search endpoint allows user to quickly search for STAC Items that meet their specific spatial, temporal, and data product requirements. Now that you learned how to navigate and explore the CMR-STAC catalog, check out the HLS_tutorial to learn how to interact with HLS data specifically.\n\n\nContact Information\nMaterial written by Mahsa Jami1 and Aaron Friesz1\nContact: LPDAAC@usgs.gov\nVoice: +1-866-573-3222\nOrganization: Land Processes Distributed Active Archive Center (LP DAAC)\nWebsite: https://lpdaac.usgs.gov/ \nDate last modified: 09-13-2021\n1 KBR, Inc., contractor to the U.S. Geological Survey, Earth Resources Observation and Science (EROS) Center, Sioux Falls, South Dakota, USA. Work performed under USGS contract G0121D0001 for LP DAAC2.\n2 LP DAAC Work performed under NASA contract NNG14HH33I." + "objectID": "environment-setup/index.html#setting-up-corn-locally", + "href": "environment-setup/index.html#setting-up-corn-locally", + "title": "Cloud Environment Setup", + "section": "Setting up corn locally", + "text": "Setting up corn locally\nSetting up corn involves two steps: (1) Downloading the environment.yml, and (2) Setting up the environment using a package manager ( e.g. Anaconda Navigator, mamba, conda, etc.)\n\nDownload corn environment.yml\n\nNavigate to the corn GitHub repository page that contains the environment.yml file: https://github.com/NASA-Openscapes/corn/blob/main/ci/environment.yml\nTo download, on the file page, click the Raw button.\nSave as environment.yml\n\n\n\nUsing Anaconda Navigator\n\nOpen Anaconda Navigator\nImport environment.yml file\nName your environment something like nasaopenscapes_env [your unique name for this environment]\nValidating\n\nmore coming soon\nTODO - from local machine how will you connect to AWS?\nCloud Primer for Amazon Web Services from NASA EOSDIS" }, { - "objectID": "in-development/earthdata-python-r-handoff.html", - "href": "in-development/earthdata-python-r-handoff.html", - "title": "earthdata: Python-R Handoff", + "objectID": "LICENSE.html", + "href": "LICENSE.html", + "title": "LICENSE", "section": "", - "text": "Create once, use often: using earthdata python package for NASA Earthdata authorization and identifying the s3 links (i.e. the locations where the data are stored on Amazon Web Services), then passing those python objects to R through Quarto for analysis by R folks. These notes are a work-in-progress by Julie and Luis and we’ll tidy them up as we develop them further.\nNote: this dream is currently not working but we are sharing our progress." + "text": "Adapted from The Turing Way\nOpenscapes makes all of their materials publicly available under open source licenses.\nThe process documents and data are made available under a CC-BY license. Software are made available under an MIT license.\nThe license text listed below (describing both CC-BY and MIT licenses as well as their usage in Openscapes) is re-used under a CC-BY license from The Carpentries community materials. (Specifically from the Reproducible Science Curriculum).\n\n\nAll documentation and chapter materials in this repository are made available under the Creative Commons Attribution license. The following is a human-readable summary of (and not a substitute for) the full legal text of the CC BY 4.0 license.\nYou are free:\n\nto Share—copy and redistribute the material in any medium or format\nto Adapt—remix, transform, and build upon the material\n\nfor any purpose, even commercially.\nThe licensor cannot revoke these freedoms as long as you follow the license terms.\nUnder the following terms:\n\nAttribution—You must give appropriate credit (mentioning that your work is derived from work that is Copyright © Openscapes and, where practical, linking to https://github.com/openscapes/approach-guide), provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.\nNo additional restrictions—You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.\n\nWith the understanding that:\n\nYou do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation.\nNo warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.\n\n\n\n\nExcept where otherwise noted, the software and infrastructure provided by the The Turing Way Community are made available under the OSI-approved MIT license.\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\nTHE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE." }, { - "objectID": "in-development/earthdata-python-r-handoff.html#the-dream", - "href": "in-development/earthdata-python-r-handoff.html#the-dream", - "title": "earthdata: Python-R Handoff", + "objectID": "LICENSE.html#process-documents-and-data", + "href": "LICENSE.html#process-documents-and-data", + "title": "LICENSE", "section": "", - "text": "Create once, use often: using earthdata python package for NASA Earthdata authorization and identifying the s3 links (i.e. the locations where the data are stored on Amazon Web Services), then passing those python objects to R through Quarto for analysis by R folks. These notes are a work-in-progress by Julie and Luis and we’ll tidy them up as we develop them further.\nNote: this dream is currently not working but we are sharing our progress." - }, - { - "objectID": "in-development/earthdata-python-r-handoff.html#python-earthdata-package-for-auth-s3-links", - "href": "in-development/earthdata-python-r-handoff.html#python-earthdata-package-for-auth-s3-links", - "title": "earthdata: Python-R Handoff", - "section": "Python: earthdata package for auth & s3 links", - "text": "Python: earthdata package for auth & s3 links\nearthdata gets me the credentials, it gets me the links based on the queries.\nIn this example, the data we want is in the Cloud. For this examples we’re using this data we identified from the Earthdata Cloud Cookbook’s Multi-File_Direct_S3_Access_NetCDF_Example, and its short_name is 'ECCO_L4_SSH_05DEG_MONTHLY_V4R4'.\n\nIdentify the s3 links\nBelow is our query, pretending that that is the data and the bounding box we want.\n\n## import DataCollections class from earthdata library\nfrom earthdata import DataGranules\n\n## To find the concept_id from the shortname that we copied: \n# short_name = 'ECCO_L4_SSH_05DEG_MONTHLY_V4R4' \n# collection = DataCollections().short_name(short_name).get()\n# [c.concept_id() for c in collection] ## this returned 'C1990404799-POCLOUD'\n\n# Then we build a Query with spatiotemporal parameters. \nGranuleQuery = DataGranules().concept_id('C1990404799-POCLOUD').bounding_box(-134.7,58.9,-133.9,59.2)\n\n## We get the metadata records from CMR\ngranules = GranuleQuery.get()\n\n## Now it's time to open our data granules list. \ns3_links = [granule.data_links(access='direct') for granule in granules] \ns3_links[0]\n\nNote that files = Store(auth).open(granules) would work for Python users but open won’t work in the R world because it will create some kind of python file handlers from fsspec.\n\n\nGet the Cloud credentials\nPrerequesite: you’ll need a functioning .netrc here. earthdata expects interactivity and that did not work here with Quarto in the RStudio IDE (and it also did not work for Julie in Jupyter notebook (June 7 2022)). So, we followed the 2021-Cloud-Hackathon’s NASA_Earthdata_Authentication, copying and pasting and running that code in a Jupyter notebook. (remember to rm .netrc beforehand!)\nThen, with a nice .netrc file, the next step is to get Cloud credentials:\n\n## import the Auth class from the earthdata library\nfrom earthdata import Auth\n\nauth = Auth().login(strategy=\"netrc\")\ncredentials = auth.get_s3_credentials(cloud_provider = \"POCLOUD\") \n\nSo now we have the s3 links and the credentials to download the links, so now we can use the tutorial in R!!\nNotes\n\nLuis will update earthdata to automatically know the cloud provider so that you don’t have to specify for example POCLOUD vs PODAAC # credentials you actually don’t want to print your credentials, we were just checking that they worked\nThe resulting JSON dictionary is what we’ll export to R, and it will be valid for 1 hour. When I run into issues, I’ll say “why is this not working”, and it’s because it’s expired in 1 hour.\nWhen we want to identify the bucket level, we’ll need to remove the name of the file. For example:\n\n<s3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_1992-01_ECCO_V4r4_latlon_0p50deg.nc> includes the filename\n<s3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/> is only the bucket\n\nExpect to run into issues with listing the files in the bucket (because maybe something is restricted or maybe you can access files but not list everything that’s inside the bucket)" - }, - { - "objectID": "in-development/earthdata-python-r-handoff.html#r-data-access-from-s3-links", - "href": "in-development/earthdata-python-r-handoff.html#r-data-access-from-s3-links", - "title": "earthdata: Python-R Handoff", - "section": "R: data access from s3 links!", - "text": "R: data access from s3 links!\nAnd now I can switch to R, if R is my preferred language.\nThe blog post Using Amazon S3 with R by Danielle Navarro is hugely informative and describes how to use the aws.s3 R package.\nFirst load libraries:\n\nlibrary(dplyr)\nlibrary(readr)\nlibrary(purrr)\nlibrary(stringr)\nlibrary(tibble)\nlibrary(aws.s3) # install.packages(\"aws.s3\")\nlibrary(reticulate)\n\nTranslate credentials from python variables (created with earthdata above) to R variables using reticulate’s py$ syntax and purr’s pluck() to isolate a variable from a list:\n\n## translate credentials from python to R, map to dataframe\ncredentials_r_list <- py$credentials #YAY!\ncredentials_r <- purrr::map_df(credentials_r_list, print)\n\n## translate s3 links from python to R, create my_bucket\ns3_links_r_list <- py$s3_links\nmy_link_list <- s3_links_r_list[1] # let's just start with one\nmy_link_chr <- purrr:::map_chr(my_link_list, paste, collapse=\"\")\n#my_link <- as_tibble(my_link_chr)\n#my_link_split <- stringr::str_split(my_link, \"/\")\n#my_bucket <- str_c(\"s3://\", my_link_split[3], my_link_split[4])\nmy_bucket <- \"s3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/\"\n\nFrom the aws.s3 documentation, set up system environment variables for AWS:\n\nSys.setenv(\"AWS_ACCESS_KEY_ID\" = credentials_r$accessKeyId,\n \"AWS_SECRET_ACCESS_KEY\" = credentials_r$secretAccessKey,\n \"AWS_DEFAULT_REGION\" = \"us-west-2\",\n \"AWS_SESSION_TOKEN\" = credentials_r$sessionToken)\n\n\n# testing by hand: Luis\nSys.setenv(\"AWS_ACCESS_KEY_ID\" = \"ASIATNGJQBXBHRPIKFFB\",\n \"AWS_SECRET_ACCESS_KEY\" = \"zbYP2fueNxLK/joDAcz678mkjjzP6fz4HUN131ID\",\n \"AWS_DEFAULT_REGION\" = \"us-west-2\")\n\nFirst let’s test Danielle’s code to see if it runs. Note to Luis: the following only works when the Sys.setenv is not set:\n\nlibrary(aws.s3)\n\nbucket_exists(\n bucket = \"s3://herbariumnsw-pds/\", \n region = \"ap-southeast-2\"\n)\n\nClient error: (403) Forbidden\n\n\n[1] FALSE\nattr(,\"x-amz-bucket-region\")\n[1] \"ap-southeast-2\"\nattr(,\"x-amz-request-id\")\n[1] \"0FQ1R57F2VHGFPDF\"\nattr(,\"x-amz-id-2\")\n[1] \"N6RPTKPN3/H9tDuKNHM2ZAcChhkkn2WpfcTzhpxC3fUmiZdNEIiu1xJsQAvFSecYIuWZ28pchQW3sAPAdVU57Q==\"\nattr(,\"content-type\")\n[1] \"application/xml\"\nattr(,\"date\")\n[1] \"Thu, 07 Jul 2022 23:11:30 GMT\"\nattr(,\"server\")\n[1] \"AmazonS3\"\n\n\nNow, see if the PODAAC bucket exists:\n\naws.s3::bucket_exists(\n bucket = \"s3://podaac-ops-cumulus-protected/\", \n region = \"us-west-2\"\n)\n\nClient error: (403) Forbidden\n\n\n[1] FALSE\nattr(,\"x-amz-bucket-region\")\n[1] \"us-west-2\"\nattr(,\"x-amz-request-id\")\n[1] \"M4T3W1JZ93M08AZB\"\nattr(,\"x-amz-id-2\")\n[1] \"hvGLWqGCRB4lLf9pD8f67OsTDulSOgqd+yLWzUTRFz2tlLPVpxHr9mSREL0bQPVyo70j0hvJp+8=\"\nattr(,\"content-type\")\n[1] \"application/xml\"\nattr(,\"date\")\n[1] \"Thu, 07 Jul 2022 23:11:30 GMT\"\nattr(,\"server\")\n[1] \"AmazonS3\"\n\n\n\nherbarium_files <- get_bucket_df(\n bucket = \"s3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/\", \n region = \"us-west-2\",\n max = 20000\n) %>% \n as_tibble()\n\nIf forbidden: - 1 hour expiration time - this bucket is not listable (or protected) (hopefully this error will be clear enough)\nIf you get the following error, it’s likely because your credentials have expired:\n\n\n\n\n\n\nImportant\n\n\n\nClient error: (403) Forbidden\n[1] FALSE\nattr(,\"x-amz-bucket-region\")\n[1] \"us-west-2\"\nattr(,\"x-amz-request-id\")\n[1] \"W2PQV030PDTGDD32\"\nattr(,\"x-amz-id-2\")\n[1] \"S8C0qzL1lAYLufzUupjqplyyS/3fWCKxIELk0OJLVHGzTOqlyhof+IPFYbaRUhmJwXQelfprYCU=\"\nattr(,\"content-type\")\n[1] \"application/xml\"\nattr(,\"date\")\n[1] \"Wed, 08 Jun 2022 03:11:16 GMT\"\nattr(,\"server\")\n[1] \"AmazonS3\"" - }, - { - "objectID": "in-development/earthdata-python-r-handoff.html#dev-notes", - "href": "in-development/earthdata-python-r-handoff.html#dev-notes", - "title": "earthdata: Python-R Handoff", - "section": "Dev notes", - "text": "Dev notes\n\nChat with Andy May 26\nMaybe have a python script that takes arguments, compiled in a way that then in MatLab you can sys.admin that python script. Then he doesn’t need to know python\nOther approach would be MatLab to re-write earthdata in MatLab\nOur dream, revised: the code should be language-agnostic" + "text": "All documentation and chapter materials in this repository are made available under the Creative Commons Attribution license. The following is a human-readable summary of (and not a substitute for) the full legal text of the CC BY 4.0 license.\nYou are free:\n\nto Share—copy and redistribute the material in any medium or format\nto Adapt—remix, transform, and build upon the material\n\nfor any purpose, even commercially.\nThe licensor cannot revoke these freedoms as long as you follow the license terms.\nUnder the following terms:\n\nAttribution—You must give appropriate credit (mentioning that your work is derived from work that is Copyright © Openscapes and, where practical, linking to https://github.com/openscapes/approach-guide), provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.\nNo additional restrictions—You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.\n\nWith the understanding that:\n\nYou do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation.\nNo warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material." }, { - "objectID": "in-development/earthdata-python-r-handoff.html#background", - "href": "in-development/earthdata-python-r-handoff.html#background", - "title": "earthdata: Python-R Handoff", - "section": "Background", - "text": "Background\nThis was Luis’ original example code, but it downloads data. The examples above access it in the cloud. From https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/earthdata-access-demo.html\nfrom earthdata import Auth, DataGranules, Store\n\n# first we authenticate with NASA EDL\nauth = Auth().login(strategy=\"netrc\")\n\n# Then we build a Query with spatiotemporal parameters\nGranuleQuery = DataGranules().concept_id(\"C1575731655-LPDAAC_ECS\").bounding_box(-134.7,58.9,-133.9,59.2)\n\n# We get the metadata records from CMR\ngranules = GranuleQuery.get()\n\n# Now it{s time to download (or open) our data granules list with get()\nfiles = Store(auth).get(granules, local_path='./data')" + "objectID": "LICENSE.html#software", + "href": "LICENSE.html#software", + "title": "LICENSE", + "section": "", + "text": "Except where otherwise noted, the software and infrastructure provided by the The Turing Way Community are made available under the OSI-approved MIT license.\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\nTHE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE." }, { "objectID": "contributing/workflow.html", @@ -763,11 +1001,25 @@ "text": "Troubleshooting\n\nError: AddrInUse\nERROR: AddrInUse: Address already in use (os error 48)\nThis error is because you had more than one instance of quarto serve going in your session. So close other command line instances that are running and try again. (If you use the R package and do quarto_serve() it will automatically make sure you only ever have 1 instance.)\n\n\nLeave/exit a virtual environment\nIn your Command Line Shell, if you want to leave your virtual environment, the command is:\ndeactivate\nThe way you tell that you are in a virtual environment: it’s named in parentheses at the beginning of your prompt:\n(.venv) (base) JLos-Macbook-Pro:earthdata-cloud-cookbook lowndes$ deactivate\n(base) JLos-Macbook-Pro:earthdata-cloud-cookbook lowndes$" }, { - "objectID": "contributing/quarto-website-tutorial.html", - "href": "contributing/quarto-website-tutorial.html", - "title": "Quarto Tutorial", + "objectID": "contributing/environment.html", + "href": "contributing/environment.html", + "title": "JupyterHub environment", "section": "", - "text": "We’ve created a tutorial on how to copy (fork or download) an existing Quarto website like this one and adapt it for your own uses. Please see: https://openscapes.github.io/quarto-website-tutorial/." + "text": "Our JupyterHub is managed by 2i2c; see background from the 2021 Cloud Hackathon [JupyterHub intro]](https://nasa-openscapes.github.io/2021-Cloud-Hackathon/clinic/jupyterhub.html).\nWe create the environment with software packages for everyone to use - this includes python packages and quarto. Our environment is called corn, developed by NSIDC software engineer Luis Lopez. Here are some notes about corn; please refer to the GitHub README for details about installation.\nWhen looking at environment.yml: - Main dependency is python. Python 3.9 has some stability “pinned” the version - Next thing: pangeo-notebook. This is not pinned to a version, whatever their latest is best. This is what makes environment run in 2i2c - The rest are optional, they’ve been added as different groups want them - It runs on linux-64 and mac-OS - side note: ~ means “don’t update past breaking versions”.\nNow getting to a point where they are conflicting and can get into problems with deprecation and failing the environment. Next strategy: trim down “core” corn environment. Then have a process & writeup for teams to copy that environment and add their additional packages, and make sure it’s the one running in 2i2c. Point to existing resources as much as possible (i.e. Turing Way, Pangeo, Pythia); link to them from here and provide our-case-specific details here." + }, + { + "objectID": "contributing/environment.html#introduction", + "href": "contributing/environment.html#introduction", + "title": "JupyterHub environment", + "section": "", + "text": "Our JupyterHub is managed by 2i2c; see background from the 2021 Cloud Hackathon [JupyterHub intro]](https://nasa-openscapes.github.io/2021-Cloud-Hackathon/clinic/jupyterhub.html).\nWe create the environment with software packages for everyone to use - this includes python packages and quarto. Our environment is called corn, developed by NSIDC software engineer Luis Lopez. Here are some notes about corn; please refer to the GitHub README for details about installation.\nWhen looking at environment.yml: - Main dependency is python. Python 3.9 has some stability “pinned” the version - Next thing: pangeo-notebook. This is not pinned to a version, whatever their latest is best. This is what makes environment run in 2i2c - The rest are optional, they’ve been added as different groups want them - It runs on linux-64 and mac-OS - side note: ~ means “don’t update past breaking versions”.\nNow getting to a point where they are conflicting and can get into problems with deprecation and failing the environment. Next strategy: trim down “core” corn environment. Then have a process & writeup for teams to copy that environment and add their additional packages, and make sure it’s the one running in 2i2c. Point to existing resources as much as possible (i.e. Turing Way, Pangeo, Pythia); link to them from here and provide our-case-specific details here." + }, + { + "objectID": "contributing/environment.html#more-background", + "href": "contributing/environment.html#more-background", + "title": "JupyterHub environment", + "section": "More background", + "text": "More background\nSome articles about Python environments with Conda that are interesting and related to our 2i2c workflow:\n\nhttps://www.anaconda.com/blog/understanding-and-improving-condas-performance\nhttps://medium.com/@marius.v.niekerk/conda-metachannel-f962241c9437\nhttps://wolfv.medium.com/making-conda-fast-again-4da4debfb3b7\nhttps://pythonspeed.com/articles/conda-dependency-management/\n\nWe are doing what article the fourth article describes. We are using conda-lock with versioned dependencies. Since corn was designed to run in 2i2c I never bothered to test it for windows or OSx but I think we should offer that possibility. I added OSX and will need some volunteers to test it on Windows." }, { "objectID": "contributing/calendar-project-management.html", @@ -791,1502 +1043,1131 @@ "text": "How are we using it?\nHere we’ll use the workshop-planning repo as an example. The way we use it will evolve to meet people’s needs and use cases.\n\nIssues\nAre you leading a workshop or attending a meeting? Open an issue in the workshop-planning repo. The issue title should have the workshop name and date. Add the issue to the MainPlanning project. We can do this either from the issue or from the project.\n\n\nLabels\nAdd one or more labels to the issue, like workshop, or the names of the participating DAACs.\n\n\n\n\n\n\nLabels can be edited\n\n\n\nLabels are defined at the repo level. If you want to add a new label, check what already exists in other repos and add a new label by copying label names and color codes. (TODO: customize NASA-Openscapes org default labels - how to)\n\n\n\n\nViewing a Project\nWe’ve set up two ways to view the MainPlanning Project: as a Status Table (https://github.com/orgs/NASA-Openscapes/projects/7/views/1) or a Roadmap Schedule (https://github.com/orgs/NASA-Openscapes/projects/7/views/3).\nNotice the URLs above. Each links to a specific view of the same Project. Anyone can add a new view, sort the issues to suit their needs, name it, and save it. It will have its own URL.\n\n\nStart and End dates\nTODO: how to add and view" }, { - "objectID": "contributing/troubleshooting.html", - "href": "contributing/troubleshooting.html", - "title": "Troubleshooting", + "objectID": "contributing/onboarding.html", + "href": "contributing/onboarding.html", + "title": "Onboarding Mentors", "section": "", - "text": "Starting advice from https://nasa-openscapes.github.io/2021-Cloud-Hackathon/logistics/github-workflows.html#git-update-revert-etc" - }, - { - "objectID": "contributing/troubleshooting.html#jupyterhub", - "href": "contributing/troubleshooting.html#jupyterhub", - "title": "Troubleshooting", - "section": "JupyterHub", - "text": "JupyterHub\nNot a git repository - in your terminal if you see the following, you likely need to cd change directory into your GitHub folder.\nfatal: not a git repository (or any parent up to mount point /home)\nStopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set)." + "text": "We expect many contributors are folks who are part of the NASA Openscapes Mentor Community, involved in user support services at the NASA EOSDIS Distributed Active Archive Centers (DAACs). This page is onboarding information for these Mentors. Further chapters describe how to onboard to all the tech and workflows that everyone can use to contribute to the Cookbook." }, { - "objectID": "contributing/troubleshooting.html#git-flight-rules", - "href": "contributing/troubleshooting.html#git-flight-rules", - "title": "Troubleshooting", - "section": "Git Flight Rules", - "text": "Git Flight Rules\n“A guide for astronauts (now, programmers using Git) about what to do when things go wrong.” This is quite a comprehensive and well-maintained list of git commands, using the command line.\nhttps://github.com/k88hudson/git-flight-rules" + "objectID": "contributing/onboarding.html#welcome-to-the-mentor-community", + "href": "contributing/onboarding.html#welcome-to-the-mentor-community", + "title": "Onboarding Mentors", + "section": "Welcome to the mentor community!", + "text": "Welcome to the mentor community!\nAs we onboard more folks to grow our cross-DAAC NASA Mentors community, we will be documenting the onboarding process here. We expect Mentors to have different experiences, expertise, technical skills, and job responsibilities. Goals of onboarding will be:\n\nget to know each other, strengthen relationships across DAACs and NASA\nshare previous work by NASA Mentors\nidentify common needs and priorities\ndevelop common approaches and solutions\nstrengthen technical skills\nstrengthen teaching skills\nbuild habits and confidence contributing to common materials\nmeet Champions and other researchers to better design how we support them\n\nWe meet remotely, primarily through Mentor Calls (focus: learning and planning) and Coworking Calls (focus: screensharing and doing). We teach together through events like Hackathons, Workshops, and Openscapes Champions Cohorts." }, { - "objectID": "contributing/troubleshooting.html#git-update-revert-etc", - "href": "contributing/troubleshooting.html#git-update-revert-etc", - "title": "Troubleshooting", - "section": "Git: update, revert, etc", - "text": "Git: update, revert, etc\nThese are some useful commands to revert/delete your local changes and update your fork with the most recent information from the main branch.\n\nDelete your local changes\nThere are several ways to delete your local changes if you were playing around and want to reset. Here are a few:\n\nUndo changes you’ve maybe saved or committed, but not pushed\nThis is less time and internet intensive (no new clone/download).\nIf you’ve got changes saved, but not yet staged, committed, or pushed, you’ll delete unstaged changes in the working directory with clean:\nYou’ll need to make sure you’re in the github repository (use pwd to check your present working directory and cd to change directory)\ngit clean -df\ngit checkout -- .\n\n\nBurn it all down\nYou’ll delete the whole repo that you have locally, and then reclone.\nYou’ll need to make sure you’re in the github repository (use pwd to check your present working directory and cd to change directory)\nrm -rf YOUR-REPO\nHere is a whole blog on how to go back in time (walk back changes), with conceptual diagrams, command line code, and screenshots from RStudio. https://ohi-science.org/news/github-going-back-in-time\n\n\n\nUpdate local branch with remote main branch\nIf while you’re working you would like to update your local your-branch with the most recent updates on the main branch on GitHub.com, there are several ways to do this. Here’s one.\ngit checkout your-branch\ngit fetch\ngit merge origin/main" - }, - { - "objectID": "how-tos/how_to_guide_parking_lot.html", - "href": "how-tos/how_to_guide_parking_lot.html", - "title": "How-To Guides Parking Lot", - "section": "", - "text": "How-To Guides Parking Lot\nThis is a document to capture and organize How-To Guides that do not, yet, have a place in the Cookbook. I’ve pulled examples from the Hackday Google Doc\nNot all of these how-tos may be required. Some may be redundant. Some may be too complicated and could be broken into several How-Tos. Feel free to add comments, links, notes, etc under each suggested How-To. Also suggest changes to the wording. Following the Divio documentation guide, I’ve tried to name how-tos with descriptive names that start “How do I… ?”.\nHow do I reformat data files?\nHow do I reproject and regrid?\nHow do I aggregate data?\nHow do I plot data?\nHow do I use Harmony to reproject a data file?\nHow do I search the USGS Landsat Cloud?\nHow do I access data in the USGS Landsat Cloud?\nHow do I find OPeNDAP datasets?\nHow do I open an OPeNDAP file in the cloud? Include Authentication?\nHow do I open/access an OPeNDAP file from my local machine?\nHow do I open a Zarr file in the cloud?\nHow do I open a COG (Cloud-Optimized-GeoTIFF) in the cloud?\nHow do I use Kerchunk to reformat files? needs refining\nHow do I use CMR STAC to access data in the cloud?\nHow do I use Matlab in the cloud?\nHow do I add a python package to a 2i2c environment?\nHow do I reproduce a shared cloud environment on my local machines" + "objectID": "contributing/onboarding.html#mentor-schedule", + "href": "contributing/onboarding.html#mentor-schedule", + "title": "Onboarding Mentors", + "section": "Mentor schedule", + "text": "Mentor schedule\nSee our schedule of events at https://nasa-openscapes.github.io/mentors" }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#outline", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#outline", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Outline:", - "text": "Outline:\n\nIntroduction to NASA Earthdata’s move to the cloud\n\nBackground and motivation\nEnabling Open Science via “Analysis-in-Place”\nResources for cloud adopters: NASA Earthdata Openscapes\n\nNASA Earthdata discovery and access in the cloud\n\nPart 1: Explore Earthdata cloud data availablity\nPart 2: Working with Cloud-Optimized GeoTIFFs using NASA’s Common Metadata Repository Spatio-Temporal Assett Catalog (CMR-STAC)\nPart 3: Working with Zarr-formatted data using NASA’s Harmony cloud transformation service\n\n\n\nTutorial materials are adapted from repos on the NASA Openscapes public Github:\n\nThis notebook source code: update https://github.com/NASA-Openscapes/2021-Cloud-Workshop-AGU/tree/main/how-tos\nAlso available via online Quarto book: update https://nasa-openscapes.github.io/2021-Cloud-Workshop-AGU/" + "objectID": "contributing/onboarding.html#tooling-checklist", + "href": "contributing/onboarding.html#tooling-checklist", + "title": "Onboarding Mentors", + "section": "Tooling checklist", + "text": "Tooling checklist\nThe tooling we use to collaborate include the following. We don’t expect everyone to have the same level of skills across all software. We will hold skill-building clinics to help strengthen familiarity and confidence, with the goal that everyone is able to contribute through overlapping and interoperable skills.\nWhen you join, expect to get access to the following. We’ll track this in the OpenscapesParticipantsList [ nasa-mentors ] spreadsheet, which is in the Google Drive Folder. Note that to add you to GitHub and the 2i2c JupyterHub, we will need you to share your GitHub username see setup instructions.\n\n✅ Google Group nasa-openscapes-mentors(We use this list primarily for calendar invites and occasional emails to the whole group)\n✅ Google Calendar invitations (for Cohort Calls and Coworking)\n✅ Google Drive (for shared notes and presentations; see below)\n✅ Slack (for connecting to NASA Mentors and broader Openscapes community; see below)\n✅ GitHub (for sharing code and documentation, project management and code review; see Contibuting Setup). We will ask for your GitHub username and add you to the NASA-Openscapes Organization.\n✅ 2i2c JupyterHub (for Python, R, and Matlab; see Contibuting Setup). We will add you with your GitHub username.\n✅ Carpentries Instructor Training (for a common pedagogy for teaching learners code- and data- intensive science).\n\n\nOpenscapes Slack\nChannels to join:\n\nnasa-mentors (private)\n2023-nasa-champions (private)\n#cloud-infrastructure\n\nHere is deeper documentation about Openscapes Slack.\n\n\nMentors Google Drive\nWe have a shared Google Drive folder: NASA-OpenscapesMentors [ nasa-mentors ]. We’ll add you with edit privileges. There is a README file within it that orients you to the folders within. The convention is that within the [ brackets ] is who has permission within the folder.\n\n\nMentors GitHub Organization\nWe’ll add you to our NASA-Openscapes GitHub Organization.\n\n\nCarpentries Instructor Training\nAs a professional development opportunity for Mentors, we partner with The Carpentries, a nonprofit that teaches introductory coding skills around the world. Their remote Instructor Training is not coding-specific, but it is a pedagogical approach to teaching technical topics. If you would like to sign up, we have an access code that you can use to select the date-times via the Instructor Training event calendar." }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#the-nasa-earthdata-archive-continues-to-grow", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#the-nasa-earthdata-archive-continues-to-grow", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "The NASA Earthdata archive continues to grow", - "text": "The NASA Earthdata archive continues to grow\n\n\n\nEOSDIS Data Archive" + "objectID": "contributing/onboarding.html#background", + "href": "contributing/onboarding.html#background", + "title": "Onboarding Mentors", + "section": "Background", + "text": "Background\nNASA Openscapes is a cross-DAAC mentor community creating & teaching resources to help researchers migrate workflows to the cloud. We meet and learn together, working with alignment on common needs for researchers using NASA Earthdata. Mentors receive Carpentries Instructor training and Openscapes mentorship to help you refine teaching strategies for Cloud, while collaborating on cloud tutorials to reuse rather than reinventing separately.\nTogether, we will teach and support in opportunities of different types: Hackathons, Workshops, and the Openscapes Champions program. We expect a range of technical ability and teaching experience. Through these growth opportunities within a broader community engagement approach, you will also gain visibility for your work as well as for your DAACs. The expected time commitment is 3-4 days/month per year. This accounts for a minimum of 4 hours/month of synchronous calls with the rest of the time being self-directed towards improving support approaches for your specific DAAC.\n\nResources\nTo get a broader sense of the NASA Openscapes Community’s work, please review these recent resources. View the full list of talks and workshops at https://nasa-openscapes.github.io.\n\nNASA Openscapes: Supporting NASA EarthData Users Transition to the Cloud - November 2 2023. Erin Robinson, GHRC User Working Group\nNASA Openscapes Mentors’ Retreat 2023: Summary and Moving Toward Sustainability - September 18, 2023. Catalina Taglialatela and the NASA Openscapes Mentors, NASA HQ Meeting\nWorking with NASA Earthdata in the Cloud- January 24, 2023 - Amy Steiker, Bri Lind, Julie Lowndes, Luis López, Michele Thornton, and the NASA Openscapes Mentors. [ESIP Winter Meeting](https://2023januaryesipmeeting.sched.com/event/1EwXE/enabling-open-science-with-nasas-earthdata-in-the-cloud) “Enabling Open Science with NASA’s Earthdata in the Cloud” Session\nNASA Openscapes: Movement building with the Flywheel - March 31, 2023, Erin Robinson & Julie Lowndes. NASA Open Source Science Working Group\nShifting institutional culture to develop climate solutions with Open Science Lowndes et al 2023 (preprint, EarthArXiv)" }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#the-nasa-earthdata-cloud-evolution", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#the-nasa-earthdata-cloud-evolution", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "The NASA Earthdata Cloud Evolution", - "text": "The NASA Earthdata Cloud Evolution\n \n\nNASA Distributed Active Archive Centers (DAACs) are continuing to migrate data to the Earthdata Cloud\n\nSupporting increased data volume as new, high-resolution remote sensing missions launch in the coming years\nData hosted via Amazon Web Services, or AWS\nDAACs continuing to support tools, services, and tutorial resources for our user communities" + "objectID": "leading-workshops/add-folks-to-2i2c-github-teams.html", + "href": "leading-workshops/add-folks-to-2i2c-github-teams.html", + "title": "How to Add Folks to the 2i2c Hub", + "section": "", + "text": "We use GitHub Teams to manage access to the 2i2c Hub. There are three different patterns of access:\nAccess for for cohorts and workshop participants currently uses the same workflow.\nWe are also working on a method to give just-in-time access for specific workshops without using GitHub Teams or other managed access. Details of this process will be added as we finalize it." }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#nasa-earthdata-cloud-as-an-enabler-of-open-science", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#nasa-earthdata-cloud-as-an-enabler-of-open-science", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "NASA Earthdata Cloud as an enabler of Open Science", - "text": "NASA Earthdata Cloud as an enabler of Open Science\n\nReducing barriers to large-scale scientific research in the era of “big data”\nIncreasing community contributions with hands-on engagement\nPromoting reproducible and shareable workflows without relying on local storage systems\n\n\n\n\nOpen Data" + "objectID": "leading-workshops/add-folks-to-2i2c-github-teams.html#individual-long-term-access", + "href": "leading-workshops/add-folks-to-2i2c-github-teams.html#individual-long-term-access", + "title": "How to Add Folks to the 2i2c Hub", + "section": "Individual long-term access", + "text": "Individual long-term access\nIndividuals are given long-term access by adding them as members of the LongtermAccess-2i2c team in the NASA-Openscapes organization.\nThis is the workflow to add individual users one by one.\n\n1. Share Google Form with users needing long-term access\nAsk users: Please add your GitHub Username to this Google form for NASA Openscapes JupyterHub Access. Requests also sometimes come in via email directly from NASA mentors.\n\n\n2. Google Form response setup\nSet the form responses in Openscapes_JupyterHubAccess to notify when there are responses: Click on Responses > click three vertical dots and then check “email notification.”\n\n\n\n\n\n\n\n\n\n\n\n\n3. Email alert of new signup\nWhen you get an email click ‘view response’ (not view summary)\n\n\n\n\n\n\n\n4. Copy GitHub username\nOn the response page, copy the Github Username \n\n\n5. Add GitHub username to the GitHub Team\n\n\n\n\n\n\nTip\n\n\n\nIf you are adding many new mentors to the long-term access team, you can use the same steps as for adding workshop participants\n\n\nGo to the NASA Openscapes LongtermAccess-2i2c Team in GitHub > Click Add member > Paste GitHub username. Note there are separate GitHub Teams for Champions Cohorts and workshops.\n\n\n\n\n\nGitHub may ask you for 2-factor authentication for the first one you add in a batch, not after that.\nPeople may see invitation notifications in two ways; in GitHub itself, and/or via email. They must accept the invitation to be able to access the Hub. This is what it looks like in their email inbox:\n\n\n\n\n\nThis is what the invitation looks like in the GitHub website interface - clicking the yellow banner notification leads to the invitation:\n\n\n\n\n\n\n\n\n\n\n\n\n6. Email confirmation\nGo back to the Google form response and grab their email address. Send the following emails to the person.\nEmail for General requests: \n\nSubject: NASA Openscapes Access to 2i2c Hub\nHi [FIRST NAME],\nI have added you to the NASA Openscapes 2i2c Jupyter Hub. Here is the link to the hub: https://openscapes.2i2c.cloud/ \nThere is a getting started guide in the NASA Earthdata Cloud Cookbook here: https://nasa-openscapes.github.io/earthdata-cloud-cookbook/\nWe’d love to know about the kind of work you are doing on the hub. Please add a brief description of your progress as you go at https://github.com/NASA-Openscapes/2i2cAccessPolicies/discussions/2. We will follow up in the next few months. \nBest,\n\n\n\n7. Register date in Form Response Sheet\nIn the Form Response Sheet, in column Added to NASA 2i2c hub add the date they were added to the Hub, or if the request came via email add a new row - in part so that we as admins knew if someone had already fulfilled the request. Check Column A for the date of the form request.\n\n\nDirections for invitees\n\nPlease go to https://github.com/nasa-openscapes\nClick the big yellow bar that says “you’ve been invited” \nNot seeing that bar? \n\nAre you logged into GitHub.com?\nHave you shared your GitHub username via this form?\nYou can also check your email that you use for GitHub and look for an invitation from GitHub and NASA-Openscapes" }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#data-and-analysis-co-located-in-place", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#data-and-analysis-co-located-in-place", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Data and Analysis co-located “in place”", - "text": "Data and Analysis co-located “in place”\n\n\n\nEarthdata Cloud Paradigm" + "objectID": "leading-workshops/add-folks-to-2i2c-github-teams.html#adding-workshop-participants-or-champions-cohorts-as-a-batch", + "href": "leading-workshops/add-folks-to-2i2c-github-teams.html#adding-workshop-participants-or-champions-cohorts-as-a-batch", + "title": "How to Add Folks to the 2i2c Hub", + "section": "Adding workshop participants or Champions cohorts as a batch", + "text": "Adding workshop participants or Champions cohorts as a batch\nParticipants in workshops run by NASA Openscapes Mentors are given Openscapes 2i2c JupyterHub access, as are participants in Champions Cohorts.\nWe use a dedicated GitHub Organization - nasa-openscapes-workshops - to manage access, with GitHub Teams for workshops and Champions Cohorts.\n\n1. Create a team in nasa-openscapes-workshops\nThere are several teams in this organization; the AdminTeam team is for members who have permission to create teams and add members to teams.\n\nIf this is for a one-off workshop, name the team [workshop-name]-[workshop-date], with workshop date in the format yyyy-mm-dd\nIf this is for a new champions cohort, name the team nasa-champions-yyyy\n\n\n\n2. Add team name to workshop registry\nWe keep a registry of all workshops in this sheet. Add the name of the GitHub team to 2i2c access type column in the appropriate row.\n\n\n3. Add teams to 2i2c Hub configuration\nMake a pull request to the Openscapes common.values.yaml file in the 2i2c infrastructure repository, adding the new team to the allowed-teams item in whichever sections are appropriate for your workshop/cohort (likely all of them):\n\ndisplay_name: Python\ndisplay_name: R\ndisplay_name: Matlab\ndisplay_name: \"Bring your own image\"\n\nThese will determine which images are available to participants when they log in to the Hub.\nTeam specification is in the form [github-org]:[team] (e.g., nasa-openscapes-workshops:champions-access-2i2c-2024)\nYou also need to add the team to the allowed_organizations item under hub > config > GitHubOAuthenticator\nYou can see an example pull request here.\n\n\n4. Share Google Form with participants needing access for the cohort or workshop\nAsk users: Please add your GitHub Username to this Google form for NASA Openscapes JupyterHub Access.\n\n\n5. Add users to a Team in a batch using kyber\n\n\n\n\n\n\nTip\n\n\n\nDo this fairly close to the day of the workshop to try to get the greatest number particpants to have filled out the form, and reduce the number of manual additions on the day of the workshop.\n\n\n\nOpen the spreadsheet of form responses, select the GitHub usernames from the rows containing the people you want to add, and copy it to your clipboard.\n\n\n\n\n\n\n\nTip\n\n\n\nSome workshop instructors will use their own form to record participants, rather than the Openscapes form. In that case use those responses instead. But do paste all those rows into this Main form; it is important that we have a record of everyone using the Hub and a way to contact them via email.\n\n\n\nIn R, do the following:\n\nlibrary(kyber)\nlibrary(datapasta)\n\n# Run datapasta::tribble_paste() or use the RStudio Addin 'Paste as tribble'\n\n# assign the pasted output to a new object:\nresponses <- tribble::tribble(\n # The stuff that was pasted\n)\nThe responses object will have a column of usernames. It likely won’t be called username, so you can rename it or even easier, change the name inside the tribble::tribble() call that you pasted. Verify these are the usernames you want to add, and run kyber::add_team_members() as follows:\nteam_name <- \"the name of the github team you want to add to\"\n\nadd_team_members(\n team_name, \n members = responses$username, \n org = \"nasa-openscapes-workshops\"\n)\n\nGo the appropriate team in the nasa-openscapes-workshops organization and verify that there are the appropriate “pending members”:\n\n\n\n\n\n\n\nGo back to the spreadsheet of responses, and fill in the Added to NASA 2i2c hub column with the date you added them.\n\n\n\nDirections for workshop participants\n\nPlease go to https://github.com/nasa-openscapes-workshops\nClick the big yellow bar that says “you’ve been invited” \nNot seeing that bar? \n\nAre you logged into GitHub.com?\nHave you shared your GitHub username via this form?\nYou can also check your email that you use for GitHub and look for an invitation from GitHub and NASA-Openscapes-Workshops" }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#building-nasa-earthdata-cloud-resources", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#building-nasa-earthdata-cloud-resources", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Building NASA Earthdata Cloud Resources", - "text": "Building NASA Earthdata Cloud Resources\nShow slide with 3 panels of user resources\nEmphasize that the following tutorials are short examples that were taken from the tutorial resources we have been building for our users" + "objectID": "leading-workshops/add-folks-to-2i2c-github-teams.html#troubleshooting", + "href": "leading-workshops/add-folks-to-2i2c-github-teams.html#troubleshooting", + "title": "How to Add Folks to the 2i2c Hub", + "section": "Troubleshooting", + "text": "Troubleshooting\n\nUsers get 403 error\nThis is a common issue. User must find email notification from GitHub and accept invitation. If they had clicked in email to accept but did not complete accepting in GitHub, edit their invitation in GitHub to uninvite and reinvite. They should get a new email to accept.\n\n\n\n\n\n\n\nDidn’t get email\nGo to https://github.com/nasa-openscapes-workshops, see the big yellow bar that says You’re invited. \n\n\nNotes on process and any issues\nYuvi has started documenting the issues we are running into here -> https://hackmd.io/tWfqVai4SDC1CbQ4mhfIbw" }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#nasa-earthdata-cloud-discovery-and-access-using-open-source-technologies", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#nasa-earthdata-cloud-discovery-and-access-using-open-source-technologies", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "NASA Earthdata Cloud: Discovery and access using open source technologies", - "text": "NASA Earthdata Cloud: Discovery and access using open source technologies\nThe following tutorial demonstrates several basic end-to-end workflows to interact with data “in-place” from the NASA Earthdata Cloud, accessing Amazon Web Services (AWS) Single Storage Solution (S3) data locations without the need to download data. While the data can be downloaded locally, the cloud offers the ability to scale compute resources to perform analyses over large areas and time spans, which is critical as data volumes continue to grow.\nAlthough the examples we’re working with in this notebook only focuses on a small time and area for demonstration purposes, this workflow can be modified and scaled up to suit a larger time range and region of interest.\n\nDatasets of interest:\n\nHarmonized Landsat Sentinel-2 (HLS) Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0 (L30) (10.5067/HLS/HLSL30.002)\n\nSurface reflectance (SR) and top of atmosphere (TOA) brightness data\nGlobal observations of the land every 2–3 days at 30-meter (m)\nCloud Optimized GeoTIFF (COG) format\n\nECCO Sea Surface Height - Daily Mean 0.5 Degree (Version 4 Release 4)(10.5067/ECG5D-SSH44).\n\nDaily-averaged dynamic sea surface height\nTime series of monthly NetCDFs on a 0.5-degree latitude/longitude grid." + "objectID": "index.html", + "href": "index.html", + "title": "NASA Earthdata Cloud Cookbook", + "section": "", + "text": "Welcome to the NASA Openscapes Earthdata Cloud Cookbook!\nThis Cookbook is learning-oriented to support scientific researchers who use NASA Earthdata as NASA migrates data and workflows to the cloud. It has resources to learn how to work with NASA Earthdata in the cloud, as well as documentation for how to contribute to these resources. It is under active, open development." }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#part-1-explore-data-hosted-in-the-earthdata-cloud", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#part-1-explore-data-hosted-in-the-earthdata-cloud", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Part 1: Explore Data hosted in the Earthdata Cloud", - "text": "Part 1: Explore Data hosted in the Earthdata Cloud\n\nEarthdata Search Demonstration\nFrom Earthdata Search https://search.earthdata.nasa.gov, use your Earthdata login credentials to log in. You can create an Earthdata Login account at https://urs.earthdata.nasa.gov.\nIn this example we are interested in the ECCO dataset, hosted by the PO.DAAC. This dataset is available from the NASA Earthdata Cloud archive hosted in AWS cloud.\nClick on the “Available from AWS Cloud” filter option on the left. Here, 39 matching collections were found with the ECCO monthly SSH search, and for the time period for year 2015. The latter can be done using the calendar icon on the left under the search box. Scroll down the list of returned matches until we see the dataset of interest, in this case ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4).\n\n\nView and Select Data Access Options\nClicking on the ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4) dataset, we now see a list of files (granules) that are part of the dataset (collection). We can click on the green + symbol to add a few files to our project. Here we added the first 3 listed for 2015. Then click on the green button towards the bottom that says “Download”. This will take us to another page with options to customize our download or access link(s).\n\n\n\nFigure caption: Select granules and click download" + "objectID": "index.html#welcome", + "href": "index.html#welcome", + "title": "NASA Earthdata Cloud Cookbook", + "section": "", + "text": "Welcome to the NASA Openscapes Earthdata Cloud Cookbook!\nThis Cookbook is learning-oriented to support scientific researchers who use NASA Earthdata as NASA migrates data and workflows to the cloud. It has resources to learn how to work with NASA Earthdata in the cloud, as well as documentation for how to contribute to these resources. It is under active, open development." }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#access-options", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#access-options", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Access Options", - "text": "Access Options\nSelect the “Direct Download” option to view Access options via Direct Download and from the AWS Cloud. Additional options to customize the data are also available for this dataset.\n\n\n\nFigure caption: Customize your download or access" + "objectID": "index.html#the-cloud-paradigm", + "href": "index.html#the-cloud-paradigm", + "title": "NASA Earthdata Cloud Cookbook", + "section": "The Cloud Paradigm", + "text": "The Cloud Paradigm\nNASA Earthdata is available from Earth Observing System Data and Information System (EOSDIS) Distributed Active Archive Centers (DAACs) that are in the process of moving data holdings to a cloud platform. In the new paradigm, data storage (orange in the illustration) will migrate to the cloud (green) and DAAC-provided tools and services built on top of the data are co-located in the Earthdata Cloud.\n\n\n\nIllustration by Catalina Oaida, PO.DAAC\n\n\nAs this data migration occurs, DAACs will have more information about how users can access data. For example, the Cloud Data page at PO.DAAC offers access to resources to help guide data users in discovering, accessing, and utilizing cloud data. During this transition, some data will continue to be available from the traditional on premise archive, while some data will also be available from and within the Earthdata Cloud.\nLearn More About NASA’s Earthdata Cloud Migration\n\nNASA EarthData Cloud Evolution - Starting point for understanding the background for moving to the Cloud.\nHow We Work - Nice overview of EOSDIS work and emphasis on open science by Katie Baynes, Keynote at FedGeoDay2020 (video).\nCloud Data Egress: How EOSDIS Supports User Needs - Describes supporting users’ needs during cloud migration by providing analysis-ready data by Emily Cassidy (blog)." }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#earthdata-cloud-access-information", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#earthdata-cloud-access-information", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Earthdata Cloud access information", - "text": "Earthdata Cloud access information\nClicking the green Download Data button again, will take us to the final page for instructions to download and links for data access in the cloud. The AWS S3 Access tab provides the S3:// links, which is what we would use to access the data directly in-region (us-west-2) within the AWS cloud.\nE.g.: s3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-09_ECCO_V4r4_latlon_0p50deg.nc where s3 indicates data is stored in AWS S3 storage, podaac-ops-cumulus-protected is the bucket, and ECCO_L4_SSH_05DEG_MONTHLY_V4R4 is the object prefix (the latter two are also listed in the dataset collection information under Cloud Access (step 3 above)).\n\nIntegrate file links into programmatic workflow, locally or in the AWS cloud.\nIn the next two examples we will work programmatically in the cloud to access datasets of interest, to get us set up for further scientific analysis of choice. There are several ways to do this. One way to connect the search part of the workflow we just did in Earthdata Search to our next steps working in the cloud is to simply copy/paste the s3:// links provides in Step 4 above into a JupyterHub notebook or script in our cloud workspace, and continue the data analysis from there.\nOne could also copy/paste the s3:// links and save them in a text file, then open and read the text file in the notebook or script in the JupyterHub in the cloud.\n\n\n\nFigure caption: Direct S3 access" + "objectID": "index.html#about", + "href": "index.html#about", + "title": "NASA Earthdata Cloud Cookbook", + "section": "About", + "text": "About\nThis Earthdata Cloud Cookbook is being developed as a cross-DAAC collaboration by the NASA-Openscapes Team. Learn more at nasa-openscapes.github.io." }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#part-2-working-with-cloud-optimized-geotiffs-using-nasas-common-metadata-repository-spatio-temporal-assett-catalog-cmr-stac", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#part-2-working-with-cloud-optimized-geotiffs-using-nasas-common-metadata-repository-spatio-temporal-assett-catalog-cmr-stac", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Part 2: Working with Cloud-Optimized GeoTIFFs using NASA’s Common Metadata Repository Spatio-Temporal Assett Catalog (CMR-STAC)", - "text": "Part 2: Working with Cloud-Optimized GeoTIFFs using NASA’s Common Metadata Repository Spatio-Temporal Assett Catalog (CMR-STAC)\nIn this example we will access the NASA’s Harmonized Landsat Sentinel-2 (HLS) version 2 assets, which are archived in cloud optimized geoTIFF (COG) format archived by the Land Processes (LP) DAAC. The COGs can be used like any other GeoTIFF file, but have some added features that make them more efficient within the cloud data access paradigm. These features include: overviews and internal tiling.\n\nBut first, what is STAC?\nSpatioTemporal Asset Catalog (STAC) is a specification that provides a common language for interpreting geospatial information in order to standardize indexing and discovering data.\nThe STAC specification is made up of a collection of related, yet independent specifications that when used together provide search and discovery capabilities for remote assets.\n\nFour STAC Specifications\nSTAC Catalog (aka DAAC Archive)\nSTAC Collection (aka Data Product)\nSTAC Item (aka Granule)\nSTAC API\n\n\n\nCMR-STAC API\nThe CMR-STAC API is NASA’s implementation of the STAC API specification for all NASA data holdings within EOSDIS. The current implementation does not allow for querries accross the entire NASA catalog. Users must execute searches within provider catalogs (e.g., LPCLOUD) to find the STAC Items they are searching for. All the providers can be found at the CMR-STAC endpoint here: https://cmr.earthdata.nasa.gov/stac/.\nIn this example, we will query the LPCLOUD provider to identify STAC Items from the Harmonized Landsat Sentinel-2 (HLS) collection that fall within our region of interest (ROI) and within our specified time range." + "objectID": "index.html#citation", + "href": "index.html#citation", + "title": "NASA Earthdata Cloud Cookbook", + "section": "Citation", + "text": "Citation\nPlease cite the NASA Earthdata Cloud Cookbook through the project’s Zenodo archive using DOI: 10.5281/zenodo.7786710. This DOI represents all versions, and will always resolve to the latest one.\nThe citation will look something like:\n\nAndy Barrett, Chris Battisto, Brandon Bottomley, Aaron Friesz, Alexis Hunzinger, Mahsa Jami, Alex Lewandowski, Bri Lind, Luis López, Jack McNelis, Cassie Nickles, Catalina Oaida Taglialatela, Celia Ou, Brianna Pagán, Sargent Shriver, Amy Steiker, Michele Thornton, Makhan Virdi, Jessica Nicole Welch, Jess Welch, Erin Robinson, Julia Stewart Lowndes. (2023). NASA-Openscapes/earthdata-cloud-cookbook: NASA EarthData Cloud Cookbook v2023.03.0 (v2023.03.0). Zenodo. https://doi.org/10.5281/zenodo.7786711\n\nPlease visit the Cookbook’s DOI link to get the most recent version - the one above is not automatically generated and may be out of date as we release updated versions of the Cookbook." }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#import-packages", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#import-packages", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Import packages", - "text": "Import packages\n\nimport os\nimport requests \nimport boto3\nfrom osgeo import gdal\nimport rasterio as rio\nfrom rasterio.session import AWSSession\nimport rioxarray\nimport hvplot.xarray\nimport holoviews as hv\n\nfrom pystac_client import Client \nfrom collections import defaultdict \nimport json\nimport geopandas\nimport geoviews as gv\nfrom cartopy import crs\ngv.extension('bokeh', 'matplotlib')" + "objectID": "index.html#license", + "href": "index.html#license", + "title": "NASA Earthdata Cloud Cookbook", + "section": "License", + "text": "License\nOpenscapes makes all of their materials publicly available under open source licenses.\nOur License is adapted from The Turing Way" }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#connect-to-the-cmr-stac-api", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#connect-to-the-cmr-stac-api", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Connect to the CMR-STAC API", - "text": "Connect to the CMR-STAC API\n\nSTAC_URL = 'https://cmr.earthdata.nasa.gov/stac'\n\n\nprovider_cat = Client.open(STAC_URL)\n\n\nConnect to the LPCLOUD Provider/STAC Catalog\nFor this next step we need the provider title (e.g., LPCLOUD). We will add the provider to the end of the CMR-STAC API URL (i.e., https://cmr.earthdata.nasa.gov/stac/) to connect to the LPCLOUD STAC Catalog.\n\ncatalog = Client.open(f'{STAC_URL}/LPCLOUD/')\n\nSince we are using a dedicated client (i.e., pystac-client.Client) to connect to our STAC Provider Catalog, we will have access to some useful internal methods and functions (e.g., get_children() or get_all_items()) we can use to get information from these objects." + "objectID": "appendix/appendix-alternate-in-cloud-access/Access_COG.html", + "href": "appendix/appendix-alternate-in-cloud-access/Access_COG.html", + "title": "Access Cloud-Optimized GeoTIFF", + "section": "", + "text": "We will access data for the Harmonized Landsat Sentinel-2 (HLS) Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0 (L30) ([10.5067/HLS/HLSL30.002](https://doi.org/10.5067/HLS/HLSL30.002)) data product. These data are archived and distributed as Cloud Optimized GeoTIFF (COG) files, one file for each spectral band.\nWe will access a single COG file, L30 red band (0.64 – 0.67 μm), from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataarray. This approach leverages S3 native protocols for efficient access to the data." }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#search-for-stac-items", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#search-for-stac-items", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Search for STAC Items", - "text": "Search for STAC Items\nWe will define our ROI using a geojson file containing a small polygon feature in western Nebraska, USA. We’ll also specify the data collections and a time range for our example.\n\nRead in a geojson file and plot\nReading in a geojson file with geopandas and extract coodinates for our ROI.\n\nfield = geopandas.read_file('../data/ne_w_agfields.geojson')\nfieldShape = field['geometry'][0]\nroi = json.loads(field.to_json())['features'][0]['geometry']\n\nWe can plot the polygon using the geoviews package that we imported as gv with ‘bokeh’ and ‘matplotlib’ extensions. The following has reasonable width, height, color, and line widths to view our polygon when it is overlayed on a base tile map.\n\nbase = gv.tile_sources.EsriImagery.opts(width=650, height=500)\nfarmField = gv.Polygons(fieldShape).opts(line_color='yellow', line_width=10, color=None)\nbase * farmField\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\nWe will now start to specify the search criteria we are interested in, i.e, the date range, the ROI, and the data collections, that we will pass to the STAC API." + "objectID": "appendix/appendix-alternate-in-cloud-access/Access_COG.html#summary", + "href": "appendix/appendix-alternate-in-cloud-access/Access_COG.html#summary", + "title": "Access Cloud-Optimized GeoTIFF", + "section": "", + "text": "We will access data for the Harmonized Landsat Sentinel-2 (HLS) Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0 (L30) ([10.5067/HLS/HLSL30.002](https://doi.org/10.5067/HLS/HLSL30.002)) data product. These data are archived and distributed as Cloud Optimized GeoTIFF (COG) files, one file for each spectral band.\nWe will access a single COG file, L30 red band (0.64 – 0.67 μm), from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataarray. This approach leverages S3 native protocols for efficient access to the data." }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#search-the-cmr-stac-api-with-our-search-criteria", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#search-the-cmr-stac-api-with-our-search-criteria", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Search the CMR-STAC API with our search criteria", - "text": "Search the CMR-STAC API with our search criteria\nNow we can put all our search criteria together using catalog.search from the pystac_client package. STAC Collection is synonomous with what we usually consider a NASA data product. Desired STAC Collections are submitted to the search API as a list containing the collection id. Let’s focus on S30 and L30 collections.\n\ncollections = ['HLSL30.v2.0', 'HLSS30.v2.0']\n\ndate_range = \"2021-05/2021-08\"\n\nsearch = catalog.search(\n collections=collections,\n intersects=roi,\n datetime=date_range,\n limit=100\n)\n\n\nView STAC Items that matched our search query\n\nprint('Matching STAC Items:', search.matched())\nitem_collection = search.get_all_items()\nitem_collection[0].to_dict()\n\nMatching STAC Items: 113\n\n\n{'type': 'Feature',\n 'stac_version': '1.0.0',\n 'id': 'HLS.L30.T13TGF.2021124T173013.v2.0',\n 'properties': {'datetime': '2021-05-04T17:30:13.428000Z',\n 'start_datetime': '2021-05-04T17:30:13.428Z',\n 'end_datetime': '2021-05-04T17:30:37.319Z',\n 'eo:cloud_cover': 36},\n 'geometry': {'type': 'Polygon',\n 'coordinates': [[[-101.5423534, 40.5109845],\n [-101.3056118, 41.2066375],\n [-101.2894253, 41.4919436],\n [-102.6032964, 41.5268623],\n [-102.638891, 40.5386175],\n [-101.5423534, 40.5109845]]]},\n 'links': [{'rel': 'self',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0/items/HLS.L30.T13TGF.2021124T173013.v2.0'},\n {'rel': 'parent',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0'},\n {'rel': 'collection',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0'},\n {'rel': <RelType.ROOT: 'root'>,\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/',\n 'type': <MediaType.JSON: 'application/json'>,\n 'title': 'LPCLOUD'},\n {'rel': 'provider', 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD'},\n {'rel': 'via',\n 'href': 'https://cmr.earthdata.nasa.gov/search/concepts/G2144020713-LPCLOUD.json'},\n {'rel': 'via',\n 'href': 'https://cmr.earthdata.nasa.gov/search/concepts/G2144020713-LPCLOUD.umm_json'}],\n 'assets': {'B11': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B11.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B11.tif'},\n 'B07': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B07.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B07.tif'},\n 'SAA': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.SAA.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.SAA.tif'},\n 'B06': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B06.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B06.tif'},\n 'B09': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B09.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B09.tif'},\n 'B10': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B10.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B10.tif'},\n 'VZA': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.VZA.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.VZA.tif'},\n 'SZA': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.SZA.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.SZA.tif'},\n 'B01': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B01.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B01.tif'},\n 'VAA': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.VAA.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.VAA.tif'},\n 'B05': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B05.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B05.tif'},\n 'B02': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B02.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B02.tif'},\n 'Fmask': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.Fmask.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.Fmask.tif'},\n 'B03': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B03.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B03.tif'},\n 'B04': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B04.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B04.tif'},\n 'browse': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.jpg',\n 'type': 'image/jpeg',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.jpg'},\n 'metadata': {'href': 'https://cmr.earthdata.nasa.gov/search/concepts/G2144020713-LPCLOUD.xml',\n 'type': 'application/xml'}},\n 'bbox': [-102.638891, 40.510984, -101.289425, 41.526862],\n 'stac_extensions': ['https://stac-extensions.github.io/eo/v1.0.0/schema.json'],\n 'collection': 'HLSL30.v2.0'}" + "objectID": "appendix/appendix-alternate-in-cloud-access/Access_COG.html#code", + "href": "appendix/appendix-alternate-in-cloud-access/Access_COG.html#code", + "title": "Access Cloud-Optimized GeoTIFF", + "section": "Code", + "text": "Code\nHere are our recommended approaches for accessing COG data in NASA Earthdata Cloud with code.\n\nImport Packages\n\nPythonR\n\n\nIn Python we can use the earthaccess library.\nTo install the package we’ll run this code from the command line. Note: you can run shell code directly from a Jupyter Notebook cell by adding a !: !conda install.\n# Install earthaccess\nconda install -c conda-forge earthaccess\nimport earthaccess\nimport requests\nimport os\nimport boto3\nfrom osgeo import gdal\nimport rasterio as rio\nfrom rasterio.session import AWSSession\nimport rioxarray\nimport hvplot.xarray\nimport holoviews as hv\n\n#From Mahsa's tutorial in main:\n#import os\n#from osgeo import gdal\n#import rasterio as rio\n#import rioxarray\n#import hvplot.xarray\n#import holoviews as hv\n\n\nlibrary(rgdal)\nlibrary(raster)\nlibrary(terra)" }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#filtering-stac-items", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#filtering-stac-items", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Filtering STAC Items", - "text": "Filtering STAC Items\nBelow we will loop through and filter the item_collection by a specified cloud cover as well as extract the band we’d need to do an Enhanced Vegetation Index (EVI) calculation for a future analysis. We will also specify the STAC Assets (i.e., bands/layers) of interest for both the S30 and L30 collections (also in our collections variable above) and print out the first ten links, converted to s3 locations:\n\ncloudcover = 25\n\ns30_bands = ['B8A', 'B04', 'B02', 'Fmask'] # S30 bands for EVI calculation and quality filtering -> NIR, RED, BLUE, Quality \nl30_bands = ['B05', 'B04', 'B02', 'Fmask'] # L30 bands for EVI calculation and quality filtering -> NIR, RED, BLUE, Quality \n\nevi_band_links = []\n\nfor i in item_collection:\n if i.properties['eo:cloud_cover'] <= cloudcover:\n if i.collection_id == 'HLSS30.v2.0':\n #print(i.properties['eo:cloud_cover'])\n evi_bands = s30_bands\n elif i.collection_id == 'HLSL30.v2.0':\n #print(i.properties['eo:cloud_cover'])\n evi_bands = l30_bands\n\n for a in i.assets:\n if any(b==a for b in evi_bands):\n evi_band_links.append(i.assets[a].href)\n \ns3_links = [l.replace('https://data.lpdaac.earthdatacloud.nasa.gov/', 's3://') for l in evi_band_links]\ns3_links[:10]\n\n['s3://lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B04.tif',\n 's3://lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B05.tif',\n 's3://lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.Fmask.tif',\n 's3://lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B02.tif',\n 's3://lp-prod-protected/HLSL30.020/HLS.L30.T14TKL.2021133T172406.v2.0/HLS.L30.T14TKL.2021133T172406.v2.0.B02.tif',\n 's3://lp-prod-protected/HLSL30.020/HLS.L30.T14TKL.2021133T172406.v2.0/HLS.L30.T14TKL.2021133T172406.v2.0.B04.tif',\n 's3://lp-prod-protected/HLSL30.020/HLS.L30.T14TKL.2021133T172406.v2.0/HLS.L30.T14TKL.2021133T172406.v2.0.B05.tif',\n 's3://lp-prod-protected/HLSL30.020/HLS.L30.T14TKL.2021133T172406.v2.0/HLS.L30.T14TKL.2021133T172406.v2.0.Fmask.tif',\n 's3://lp-prod-protected/HLSS30.020/HLS.S30.T14TKL.2021133T173859.v2.0/HLS.S30.T14TKL.2021133T173859.v2.0.B04.tif',\n 's3://lp-prod-protected/HLSS30.020/HLS.S30.T14TKL.2021133T173859.v2.0/HLS.S30.T14TKL.2021133T173859.v2.0.B8A.tif']" + "objectID": "appendix/appendix-alternate-in-cloud-access/Access_COG.html#workspace-environment-setup", + "href": "appendix/appendix-alternate-in-cloud-access/Access_COG.html#workspace-environment-setup", + "title": "Access Cloud-Optimized GeoTIFF", + "section": "Workspace Environment Setup", + "text": "Workspace Environment Setup\n\nPythonR\n\n\nFor this exercise, we are going to open up a context manager for the notebook using the rasterio.env module to store the required GDAL configurations we need to access the data from Earthdata Cloud. While the context manager is open (rio_env.__enter__()) we will be able to run the open or get data commands that would typically be executed within a with statement, thus allowing us to more freely interact with the data. We’ll close the context (rio_env.__exit__()) at the end of the notebook.\nGDAL environment variables must be configured to access COGs from Earthdata Cloud. Geospatial data access Python packages like rasterio and rioxarray depend on GDAL, leveraging GDAL’s “Virtual File Systems” to read remote files. GDAL has a lot of environment variables that control it’s behavior. Changing these settings can mean the difference being able to access a file or not. They can also have an impact on the performance.\nrio_env = rio.Env(GDAL_DISABLE_READDIR_ON_OPEN='TRUE',\n GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/cookies.txt'),\n GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/cookies.txt'))\nrio_env.__enter__()\n\n\nSet up rgdal configurations to access the cloud assets that we are interested in. You can learn more about these configuration options here.\nsetGDALconfig(c(\"GDAL_HTTP_UNSAFESSL=YES\",\n \"GDAL_HTTP_COOKIEFILE=.rcookies\",\n \"GDAL_HTTP_COOKIEJAR=.rcookies\",\n \"GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR\",\n \"CPL_VSIL_CURL_ALLOWED_EXTENSIONS=TIF\"))\n\n\n\nIn this example we’re interested in the HLS L30 data collection from NASA’s LP DAAC in Earthdata Cloud. Below we specify the HTTPS URL to the data asset in Earthdata Cloud. This URL can be found via Earthdata Search or programmatically through the CMR and CMR-STAC APIs.\n\nPythonR\n\n\nhttps_url = 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T11SQA.2021333T181532.v2.0/HLS.L30.T11SQA.2021333T181532.v2.0.B04.tif'\n\n\nPlease note that in R, we need to add /vsicurl/ manually to the COG file URL.\nhttps_url <- '/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T11SQA.2021333T181532.v2.0/HLS.L30.T11SQA.2021333T181532.v2.0.B04.tif'" }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#access-s3-storage-location", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#access-s3-storage-location", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Access s3 storage location", - "text": "Access s3 storage location\nAccess s3 credentials from LP.DAAC and create a boto3 Session object using your temporary credentials. This Session is used to pass credentials and configuration to AWS so we can interact wit S3 objects from applicable buckets.\n\ns3_cred_endpoint = 'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials'\ntemp_creds_req = requests.get(s3_cred_endpoint).json()\n\nsession = boto3.Session(aws_access_key_id=temp_creds_req['accessKeyId'], \n aws_secret_access_key=temp_creds_req['secretAccessKey'],\n aws_session_token=temp_creds_req['sessionToken'],\n region_name='us-west-2')\n\nGDAL Configurations\nGDAL is a foundational piece of geospatial software that is leveraged by several popular open-source, and closed, geospatial software. The rasterio package is no exception. Rasterio leverages GDAL to, among other things, read and write raster data files, e.g., GeoTIFFs/Cloud Optimized GeoTIFFs. To read remote files, i.e., files/objects stored in the cloud, GDAL uses its Virtual File System API. In a perfect world, one would be able to point a Virtual File System (there are several) at a remote data asset and have the asset retrieved, but that is not always the case. GDAL has a host of configurations/environmental variables that adjust its behavior to, for example, make a request more performant or to pass AWS credentials to the distribution system. Below, we’ll identify the evironmental variables that will help us get our data from cloud\n\nrio_env = rio.Env(AWSSession(session),\n GDAL_DISABLE_READDIR_ON_OPEN='TRUE',\n GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/cookies.txt'),\n GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/cookies.txt'))\nrio_env.__enter__()\n\n<rasterio.env.Env at 0x7f64510812e0>\n\n\n\ns3_url = 's3://lp-prod-protected/HLSL30.020/HLS.L30.T11SQA.2021333T181532.v2.0/HLS.L30.T11SQA.2021333T181532.v2.0.B04.tif'\n# s3_url = 's3://lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B04.tif'" + "objectID": "appendix/appendix-alternate-in-cloud-access/Access_COG.html#https-data-access", + "href": "appendix/appendix-alternate-in-cloud-access/Access_COG.html#https-data-access", + "title": "Access Cloud-Optimized GeoTIFF", + "section": "HTTPS Data Access", + "text": "HTTPS Data Access\nRead in the HLS HTTPS URL for the L30 red band (0.64 – 0.67 μm) into our workspace. Note that, accessing files in the cloud requires you to authenticate using your NASA Earthdata Login account meaning a proper netrc file needs to be set up.\n\nPythonRMatlabCommand Line\n\n\nWe will authenticate below using a netrc file. See the (TBD) appendix for more information on netrc setup.\nauth = Auth().login(strategy=\"netrc\")\n# are we authenticated?\nif not auth.authenticated:\n # ask for credentials and persist them in a .netrc file\n auth.login(strategy=\"interactive\", persist=True)\n\n# The Store class will let us download data from NASA directly\nstore = Store(auth)\n\nWorking with the URLs directly\nIf we choose, we can use earthaccess to grab the file’s URLs and then access them with another library. Getting the links to our data is quiet simple with the data_links() method on each of the results. See the previous Find Data How-To for more information on how to discover datasets of interest.\n#Searching over a small plot in Nebraska, USA over two weeks in September 2022\ngranules = DataGranules().concept_id(\"C2021957657-LPCLOUD\").temporal(\"2022-09-10\",\"2022-09-24\").bounding_box(-101.67271,41.04754,-101.65344,41.06213).get()\nprint(len(granules))\ngranules[0].data_links(access=\"direct\")\n\n\nGet Temporary AWS Credentials\nDirect S3 access is achieved by passing NASA supplied temporary credentials to AWS so we can interact with S3 objects from applicable Earthdata Cloud buckets. For now, each NASA DAAC has different AWS credentials endpoints. Below are some of the credential endpoints to various DAACs.\nCOMING SOON: We can use the earthaccess store class to pass these credentials directly to Boto3 without the need to set up this function.\ns3_cred_endpoint = {\n 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n 'gesdisc': 'https://data.gesdisc.earthdata.nasa.gov/s3credentials',\n 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n 'ornldaac': 'https://data.ornldaac.earthdata.nasa.gov/s3credentials',\n 'ghrcdaac': 'https://data.ghrc.earthdata.nasa.gov/s3credentials'\n}\n\ndef get_temp_creds(provider):\n return requests.get(s3_cred_endpoint[provider]).json()\n\ntemp_creds_req = get_temp_creds('lpdaac')\nCreate a boto3 Session object using your temporary credentials. This Session is used to pass credentials and configuration to AWS so we can interact wit S3 objects from applicable buckets.\nsession = boto3.Session(aws_access_key_id=temp_creds_req['accessKeyId'], \n aws_secret_access_key=temp_creds_req['secretAccessKey'],\n aws_session_token=temp_creds_req['sessionToken'],\n region_name='us-west-2')\nGDAL environment variables must be configured to access COGs in Earthdata Cloud:\nrio_env = rio.Env(AWSSession(session),\n GDAL_DISABLE_READDIR_ON_OPEN='TRUE',\n GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/cookies.txt'),\n GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/cookies.txt'))\nrio_env.__enter__()\n\n\nDirect In-region Access\nRead in the HLS s3 URL for the L30 red band (0.64 – 0.67 μm) into our workspace using rioxarray, an extension of xarray used to read geospatial data. The file is read into Python as an xarray dataarray with a band, x, and y dimension. In this example the band dimension is meaningless, so we’ll use the squeeze() function to remove band as a dimension.\ns3_url = granules[0].data_links(access=\"direct\")[8]\nda = rioxarray.open_rasterio(s3_url)\nda_red = da.squeeze('band', drop=True)\nda_red\nPlot the dataarray, representing the L30 red band, using hvplot.\nda_red.hvplot.image(x='x', y='y', cmap='gray', aspect='equal')\nExit the context manager.\nrio_env.__exit__()\n\n\n\nda_red <- rast(https_url)\nda_red\nThe Convert a SpatRaster object to a Raster object using raster() to be able to use leaflet to plot our data.\nred_raster <- da_red %>% raster()\nred_raster\nThen plot the red band using plot function.\nplot(red_raster)\n\n\nMatlab code coming soon!\n#| echo: true\n# Coming soon!\n\n\nWith wget and curl:\n# Coming soon!" }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#read-cloud-optimized-geotiff-into-rioxarray", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#read-cloud-optimized-geotiff-into-rioxarray", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Read Cloud-Optimized GeoTIFF into rioxarray", - "text": "Read Cloud-Optimized GeoTIFF into rioxarray\n\nda = rioxarray.open_rasterio(s3_url)\nda\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (band: 1, y: 3660, x: 3660)>\n[13395600 values with dtype=int16]\nCoordinates:\n * band (band) int64 1\n * x (x) float64 7e+05 7e+05 7e+05 ... 8.097e+05 8.097e+05 8.097e+05\n * y (y) float64 4.1e+06 4.1e+06 4.1e+06 ... 3.99e+06 3.99e+06\n spatial_ref int64 0\nAttributes:\n _FillValue: -9999.0\n scale_factor: 0.0001\n add_offset: 0.0\n long_name: Redxarray.DataArrayband: 1y: 3660x: 3660...[13395600 values with dtype=int16]Coordinates: (4)band(band)int641array([1])x(x)float647e+05 7e+05 ... 8.097e+05 8.097e+05array([699975., 700005., 700035., ..., 809685., 809715., 809745.])y(y)float644.1e+06 4.1e+06 ... 3.99e+06array([4100025., 4099995., 4099965., ..., 3990315., 3990285., 3990255.])spatial_ref()int640crs_wkt :PROJCS[\"UTM Zone 11, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not_specified_based_on_WGS_84_spheroid\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-117],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not_specified_based_on_WGS_84_spheroidprojected_crs_name :UTM Zone 11, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-117.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 11, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not_specified_based_on_WGS_84_spheroid\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-117],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :699960.0 30.0 0.0 4100040.0 0.0 -30.0array(0)Attributes: (4)_FillValue :-9999.0scale_factor :0.0001add_offset :0.0long_name :Red\n\n\nWhen GeoTIFFS/Cloud Optimized GeoTIFFS are read in, a band coordinate variable is automatically created (see the print out above). In this exercise we will not use that coordinate variable, so we will remove it using the squeeze() function to avoid confusion.\n\nda_red = da.squeeze('band', drop=True)\nda_red\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (y: 3660, x: 3660)>\n[13395600 values with dtype=int16]\nCoordinates:\n * x (x) float64 7e+05 7e+05 7e+05 ... 8.097e+05 8.097e+05 8.097e+05\n * y (y) float64 4.1e+06 4.1e+06 4.1e+06 ... 3.99e+06 3.99e+06\n spatial_ref int64 0\nAttributes:\n _FillValue: -9999.0\n scale_factor: 0.0001\n add_offset: 0.0\n long_name: Redxarray.DataArrayy: 3660x: 3660...[13395600 values with dtype=int16]Coordinates: (3)x(x)float647e+05 7e+05 ... 8.097e+05 8.097e+05array([699975., 700005., 700035., ..., 809685., 809715., 809745.])y(y)float644.1e+06 4.1e+06 ... 3.99e+06array([4100025., 4099995., 4099965., ..., 3990315., 3990285., 3990255.])spatial_ref()int640crs_wkt :PROJCS[\"UTM Zone 11, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not_specified_based_on_WGS_84_spheroid\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-117],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not_specified_based_on_WGS_84_spheroidprojected_crs_name :UTM Zone 11, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-117.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 11, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not_specified_based_on_WGS_84_spheroid\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-117],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :699960.0 30.0 0.0 4100040.0 0.0 -30.0array(0)Attributes: (4)_FillValue :-9999.0scale_factor :0.0001add_offset :0.0long_name :Red" + "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html", + "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html", + "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", + "section": "", + "text": "In this notebook, we will access monthly sea surface height from ECCO V4r4 (10.5067/ECG5D-SSH44). The data are provided as a time series of monthly netCDFs on a 0.5-degree latitude/longitude grid.\nWe will access the data from inside the AWS cloud (us-west-2 region, specifically) and load a time series made of multiple netCDF datasets into an xarray dataset. This approach leverages S3 native protocols for efficient access to the data." }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#plot-using-hvplot", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#plot-using-hvplot", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Plot using hvplot", - "text": "Plot using hvplot\n\nda_red.hvplot.image(x='x', y='y', cmap='gray', aspect='equal')\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\nrio_env.__exit__()" + "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#summary", + "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#summary", + "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", + "section": "", + "text": "In this notebook, we will access monthly sea surface height from ECCO V4r4 (10.5067/ECG5D-SSH44). The data are provided as a time series of monthly netCDFs on a 0.5-degree latitude/longitude grid.\nWe will access the data from inside the AWS cloud (us-west-2 region, specifically) and load a time series made of multiple netCDF datasets into an xarray dataset. This approach leverages S3 native protocols for efficient access to the data." }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#part-3-working-with-zarr-formatted-data-using-nasas-harmony-cloud-transformation-service", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#part-3-working-with-zarr-formatted-data-using-nasas-harmony-cloud-transformation-service", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Part 3: Working with Zarr-formatted data using NASA’s Harmony cloud transformation service", - "text": "Part 3: Working with Zarr-formatted data using NASA’s Harmony cloud transformation service\nWe have already explored direct access to the NASA EOSDIS archive in the cloud via the Amazon Simple Storage Service (S3). In addition to directly accessing the files archived and distributed by each of the NASA DAACs, many datasets also support services that allow us to customize the data via subsetting, reformatting, reprojection, and other transformations.\nThis example demonstrates “analysis in place” using customized ECCO Level 4 monthly sea surface height data, in this case reformatted to Zarr, from a new ecosystem of services operating within the NASA Earthdata Cloud: NASA Harmony:\n\nConsistent access patterns to EOSDIS holdings make cross-data center data access easier\nData reduction services allow us to request only the data we want, in the format and projection we want\nAnalysis Ready Data and cloud access will help reduce time-to-science\nCommunity Development helps reduce the barriers for re-use of code and sharing of domain knowledge" + "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#requirements", + "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#requirements", + "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", + "section": "Requirements", + "text": "Requirements\n\n1. AWS instance running in us-west-2\nNASA Earthdata Cloud data in S3 can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region.\n\n\n2. Earthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\n3. netrc File\nYou will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. For additional information see: Authentication for NASA Earthdata." }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#import-packages-1", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#import-packages-1", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Import packages", - "text": "Import packages\n\nfrom harmony import BBox, Client, Collection, Request, LinkType\nfrom harmony.config import Environment\nfrom pprint import pprint\nimport datetime as dt\nimport s3fs\nfrom pqdm.threads import pqdm\nimport xarray as xr" + "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#learning-objectives", + "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#learning-objectives", + "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", + "section": "Learning Objectives", + "text": "Learning Objectives\n\nhow to retrieve temporary S3 credentials for in-region direct S3 bucket access\nhow to define a dataset of interest and find netCDF files in S3 bucket\nhow to perform in-region direct access of ECCO_L4_SSH_05DEG_MONTHLY_V4R4 data in S3\nhow to plot the data" }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#using-harmony-py-to-customize-data", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#using-harmony-py-to-customize-data", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Using Harmony-Py to customize data", - "text": "Using Harmony-Py to customize data\nHarmony-Py provides a pip installable Python alternative to directly using Harmony’s RESTful API to make it easier to request data and service options, especially when interacting within a Python Jupyter Notebook environment.\n\nCreate Harmony Client object\nFirst, we need to create a Harmony Client, which is what we will interact with to submit and inspect a data request to Harmony, as well as to retrieve results.\n\nharmony_client = Client()" + "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#import-packages", + "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#import-packages", + "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", + "section": "Import Packages", + "text": "Import Packages\n\nimport os\nimport requests\nimport s3fs\nimport xarray as xr\nimport hvplot.xarray" }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#create-harmony-request", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#create-harmony-request", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Create Harmony Request", - "text": "Create Harmony Request\nSpecify a temporal range over 2015, and Zarr as an output format.\nZarr is an open source library for storing N-dimensional array data. It supports multidimensional arrays with attributes and dimensions similar to NetCDF4, and it can be read by XArray. Zarr is often used for data held in cloud object storage (like Amazon S3), because it is better optimized for these situations than NetCDF4.\n\nshort_name = 'ECCO_L4_SSH_05DEG_MONTHLY_V4R4'\n\nrequest = Request(\n collection=Collection(id=short_name),\n temporal={\n 'start': dt.datetime(2015, 1, 2),\n 'stop': dt.datetime(2015, 12, 31),\n },\n format='application/x-zarr'\n)\n\njob_id = harmony_client.submit(request)" + "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#get-temporary-aws-credentials", + "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#get-temporary-aws-credentials", + "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", + "section": "Get Temporary AWS Credentials", + "text": "Get Temporary AWS Credentials\nDirect S3 access is achieved by passing NASA supplied temporary credentials to AWS so we can interact with S3 objects from applicable Earthdata Cloud buckets. For now, each NASA DAAC has different AWS credentials endpoints. Below are some of the credential endpoints to various DAACs:\n\ns3_cred_endpoint = {\n 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n 'gesdisc': 'https://data.gesdisc.earthdata.nasa.gov/s3credentials',\n 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n 'ornldaac': 'https://data.ornldaac.earthdata.nasa.gov/s3credentials',\n 'ghrcdaac': 'https://data.ghrc.earthdata.nasa.gov/s3credentials'\n}\n\nCreate a function to make a request to an endpoint for temporary credentials. Remember, each DAAC has their own endpoint and credentials are not usable for cloud data from other DAACs.\n\ndef get_temp_creds(provider):\n return requests.get(s3_cred_endpoint[provider]).json()\n\n\ntemp_creds_req = get_temp_creds('podaac')\n#temp_creds_req" }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#check-request-status-and-view-output-urls", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#check-request-status-and-view-output-urls", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Check request status and view output URLs", - "text": "Check request status and view output URLs\nHarmony data outputs can be accessed within the cloud using the s3 URLs and AWS credentials provided in the Harmony job response:\n\nharmony_client.wait_for_processing(job_id, show_progress=True)\n\nresults = harmony_client.result_urls(job_id, link_type=LinkType.s3)\ns3_urls = list(results)\ns3_urls\n\n [ Processing: 83% ] |########################################## | [/]\n\n\n\nAWS credential retrieval\nUsing aws_credentials you can retrieve the credentials needed to access the Harmony s3 staging bucket and its contents.\n\ncreds = harmony_client.aws_credentials()" + "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#set-up-an-s3fs-session-for-direct-access", + "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#set-up-an-s3fs-session-for-direct-access", + "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", + "section": "Set up an s3fs session for Direct Access", + "text": "Set up an s3fs session for Direct Access\ns3fs sessions are used for authenticated access to s3 bucket and allows for typical file-system style operations. Below we create session by passing in the temporary credentials we recieved from our temporary credentials endpoint.\n\nfs_s3 = s3fs.S3FileSystem(anon=False, \n key=temp_creds_req['accessKeyId'], \n secret=temp_creds_req['secretAccessKey'], \n token=temp_creds_req['sessionToken'],\n client_kwargs={'region_name':'us-west-2'})\n\nIn this example we’re interested in the ECCO data collection from NASA’s PO.DAAC in Earthdata Cloud. In this case it’s the following string that unique identifies the collection of monthly, 0.5-degree sea surface height data (ECCO_L4_SSH_05DEG_MONTHLY_V4R4).\n\nshort_name = 'ECCO_L4_SSH_05DEG_MONTHLY_V4R4'\n\n\nbucket = os.path.join('podaac-ops-cumulus-protected/', short_name, '*2015*.nc')\nbucket\n\nGet a list of netCDF files located at the S3 path corresponding to the ECCO V4r4 monthly sea surface height dataset on the 0.5-degree latitude/longitude grid, for year 2015.\n\nssh_files = fs_s3.glob(bucket)\nssh_files" }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#open-staged-files-with-s3fs-and-xarray", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#open-staged-files-with-s3fs-and-xarray", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Open staged files with s3fs and xarray", - "text": "Open staged files with s3fs and xarray\nAccess AWS credentials for the Harmony bucket, and use the AWS s3fs package to create a file system that can then be read by xarray. Below we create session by passing in the temporary credentials we recieved from our temporary credentials endpoint.\n\ncreds = harmony_client.aws_credentials()\n\ns3_fs = s3fs.S3FileSystem(\n key=creds['aws_access_key_id'],\n secret=creds['aws_secret_access_key'],\n token=creds['aws_session_token'],\n client_kwargs={'region_name':'us-west-2'},\n)\n\nOpen the Zarr stores using the s3fs package, then load them all at once into a concatenated xarray dataset:\n\nstores = [s3fs.S3Map(root=url, s3=s3_fs, check=False) for url in s3_urls]\ndef open_zarr_xarray(store):\n return xr.open_zarr(store=store, consolidated=True)\n\ndatasets = pqdm(stores, open_zarr_xarray, n_jobs=12)\n\nds = xr.concat(datasets, 'time', coords='minimal', )\nds = xr.decode_cf(ds, mask_and_scale=True, decode_coords=True)\nds\n\n\nssh_da = ds.SSH\n\nssh_da.to_masked_array(copy=False)\n\nssh_da" + "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#direct-in-region-access", + "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#direct-in-region-access", + "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", + "section": "Direct In-region Access", + "text": "Direct In-region Access\nOpen with the netCDF files using the s3fs package, then load them all at once into a concatenated xarray dataset.\n\nfileset = [fs_s3.open(file) for file in ssh_files]\n\nCreate an xarray dataset using the open_mfdataset() function to “read in” all of the netCDF4 files in one call.\n\nssh_ds = xr.open_mfdataset(fileset,\n combine='by_coords',\n mask_and_scale=True,\n decode_cf=True,\n chunks='auto')\nssh_ds\n\nGet the SSH variable as an xarray dataarray\n\nssh_da = ssh_ds.SSH\nssh_da\n\nPlot the SSH time series using hvplot\n\nssh_da.hvplot.image(y='latitude', x='longitude', cmap='Viridis',).opts(clim=(ssh_da.attrs['valid_min'][0],ssh_da.attrs['valid_max'][0]))" }, { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#plot-the-sea-surface-height-time-series-using-hvplot", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#plot-the-sea-surface-height-time-series-using-hvplot", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Plot the Sea Surface Height time series using hvplot", - "text": "Plot the Sea Surface Height time series using hvplot\nNow we can start looking at aggregations across the time dimension. In this case, plot the standard deviation of the temperature at each point to get a visual sense of how much temperatures fluctuate over the course of the month.\n\nssh_da = ds.SSH\n\nstdev_ssh = ssh_da.std('time')\nstdev_ssh.name = 'stdev of analysed_sst [Kelvin]'\nstdev_ssh.plot();\n\nssh_da.hvplot.image(x='longitude', y='latitude', cmap='Spectral_r', aspect='equal').opts(clim=(ssh_da.attrs['valid_min'],ssh_da.attrs['valid_max']))" - }, - { - "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#further-resources", - "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#further-resources", - "title": "AGU 2021 Open Science in Action Tutorials:", - "section": "Further Resources", - "text": "Further Resources\n\nReference Hackathon/workshop tutorials that go into more detail!\nEarthdata Cloud Cookbook\nEarthdata Cloud Primer\n\nGetting started with Amazon Web Services outside of the Workshop to access and work with data with a cloud environment." + "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#resources", + "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#resources", + "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", + "section": "Resources", + "text": "Resources\nDirect access to ECCO data in S3 (from us-west-2)\nData_Access__Direct_S3_Access__PODAAC_ECCO_SSH using CMR-STAC API to retrieve S3 links" }, { - "objectID": "how-tos/access-data/access-local-opendap.html", - "href": "how-tos/access-data/access-local-opendap.html", - "title": "How do I access Cloud Data from my Local Machine with OPeNDAP?", + "objectID": "appendix/authentication.html", + "href": "appendix/authentication.html", + "title": "Authentication", "section": "", - "text": "OPeNDAP, or the Open-source Project for a Network Data Access Protocol, is a data server that allows for accessing scientific datasets through the public internet. OPeNDAP links can be acquired through dataset landing pages or the common metadata repository. Once you have the desired link(s), you can open them as follows:\n#Import packages\nimport xarray as xr\n\nds_https = xr.open_dataset(https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_hi_tmax_2010.nc)\n\n# For datasets that contain dap4 specific datatypes, such as `int64`, replace `https` with `dap4` at the beginning to open with xarray. For example:\nurl = https://opendap.earthdata.nasa.gov/collections/C2036881966-POCLOUD/granules/AQUA_MODIS.20220109_20220116.L3m.8D.SST4.sst4.4km\ndap4_url = url.replace(\"https://\", \"dap4://\")\n# dap4_url output: dap4://opendap.earthdata.nasa.gov/collections/C2036881966-POCLOUD/granules/AQUA_MODIS.20220109_20220116.L3m.8D.SST4.sst4.4km\nds_dap4 = xr.open_dataset(dap4_url)\n\n# Note: Some datasets have their variables organized in groups, so within the open_dataset function, you may need to specify 'group=' to access the dataset.\nFor more information, see the OPeNDAP Data Access Client Tutorials." + "text": "An Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\nWe recommend authentication using earthaccess Python package, which you can also call from R. Please check the how do I find data using code on how to utilize the earthaccess package.\nAuthentication is also possible without the earthaccess package. You will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. The following code creates a hidden .netrc file (_netrc for Windows OS) with Earthdata login credentials in your home directory. This file is needed to access NASA Earthdata assets from scripting environments like Python and R. Below is code describing this process.\n\nPythonR\n\n\nPlease see this 2021 Cloud Hackathon tutorial for Python code.\n\n\nFor authentication in R, please run the code below:\n\n\n\n[R code]\n\n# Required packages for this script\npackages = c('sys', 'getPass')\n\n# Identify missing (not installed) packages\nnew.packages = packages[!(packages %in% installed.packages()[,\"Package\"])]\n\n# Install missing packages\nif(length(new.packages)) install.packages(new.packages, repos='http://cran.rstudio.com/')\n\n# Load packages into R\nlibrary(sys)\nlibrary(getPass)\n\n# Determine OS and associated netrc file \nnetrc_type <- if(.Platform$OS.type == \"windows\") \"_netrc\" else \".netrc\" # Windows OS uses _netrc file\n\n# Specify path to user profile \nup <- file.path(Sys.getenv(\"USERPROFILE\")) # Retrieve user directory (for netrc file)\n\n# Below, HOME and Userprofile directories are set. \n\nif (up == \"\") {\n up <- Sys.getenv(\"HOME\") \n Sys.setenv(\"userprofile\" = up)\n if (up == \"\") {\n cat('USERPROFILE/HOME directories need to be set up. Please type sys.setenv(\"HOME\" = \"YOURDIRECTORY\") or sys.setenv(\"USERPROFILE\" = \"YOURDIRECTORY\") in your console and type your USERPROFILE/HOME direcory instead of \"YOURDIRECTORY\". Next, run the code chunk again.')\n }\n} else {Sys.setenv(\"HOME\" = up)} \n\nnetrc_path <- file.path(up, netrc_type, fsep = .Platform$file.sep) # Path to netrc file\n\n# Create a netrc file if one does not exist already\nif (file.exists(netrc_path) == FALSE || grepl(\"urs.earthdata.nasa.gov\", readLines(netrc_path)) == FALSE) {\n netrc_conn <- file(netrc_path)\n \n # User will be prompted for NASA Earthdata Login Username and Password below\n writeLines(c(\"machine urs.earthdata.nasa.gov\",\n sprintf(\"login %s\", getPass(msg = \"Enter NASA Earthdata Login Username \\n (An account can be Created at urs.earthdata.nasa.gov):\")),\n sprintf(\"password %s\", getPass(msg = \"Enter NASA Earthdata Login Password:\"))), netrc_conn)\n close(netrc_conn)\n}else{\n i <- 0 \n for (f in readLines(netrc_path)){\n i <- i + 1\n if (f ==\"machine urs.earthdata.nasa.gov\"){\n username <- strsplit(readLines(netrc_path)[i+1], \" \")[[1]][2]\n un <- getPass(msg = paste0(\"Is your NASA Earthdata Login Username: \", username, \"\\n\\n Type yes or no.\"))\n if (tolower(un) == 'yes'){\n tx <- gsub(readLines(netrc_path)[i+2], sprintf(\"password %s\", getPass(msg = \"Enter NASA Earthdata Login Password:\")), readLines(netrc_path))\n writeLines(tx, netrc_path)\n rm(username, un, tx, f, i)\n }else{\n user <- gsub(readLines(netrc_path)[i+1], sprintf(\"login %s\", getPass(msg = \"Enter NASA Earthdata Login Username:\")), readLines(netrc_path))\n tx <- gsub(readLines(netrc_path)[i+2], sprintf(\"password %s\", getPass(msg = \"Enter NASA Earthdata Login Password:\")), readLines(netrc_path))\n writeLines(tx, netrc_path)\n rm(username, un, user, tx, f, i)\n \n }\n break\n }\n }\n}" }, { - "objectID": "how-tos/access-data/access-local-opendap.html#opendap", - "href": "how-tos/access-data/access-local-opendap.html#opendap", - "title": "How do I access Cloud Data from my Local Machine with OPeNDAP?", + "objectID": "appendix/authentication.html#earthdata-login", + "href": "appendix/authentication.html#earthdata-login", + "title": "Authentication", "section": "", - "text": "OPeNDAP, or the Open-source Project for a Network Data Access Protocol, is a data server that allows for accessing scientific datasets through the public internet. OPeNDAP links can be acquired through dataset landing pages or the common metadata repository. Once you have the desired link(s), you can open them as follows:\n#Import packages\nimport xarray as xr\n\nds_https = xr.open_dataset(https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_hi_tmax_2010.nc)\n\n# For datasets that contain dap4 specific datatypes, such as `int64`, replace `https` with `dap4` at the beginning to open with xarray. For example:\nurl = https://opendap.earthdata.nasa.gov/collections/C2036881966-POCLOUD/granules/AQUA_MODIS.20220109_20220116.L3m.8D.SST4.sst4.4km\ndap4_url = url.replace(\"https://\", \"dap4://\")\n# dap4_url output: dap4://opendap.earthdata.nasa.gov/collections/C2036881966-POCLOUD/granules/AQUA_MODIS.20220109_20220116.L3m.8D.SST4.sst4.4km\nds_dap4 = xr.open_dataset(dap4_url)\n\n# Note: Some datasets have their variables organized in groups, so within the open_dataset function, you may need to specify 'group=' to access the dataset.\nFor more information, see the OPeNDAP Data Access Client Tutorials." + "text": "An Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\nWe recommend authentication using earthaccess Python package, which you can also call from R. Please check the how do I find data using code on how to utilize the earthaccess package.\nAuthentication is also possible without the earthaccess package. You will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. The following code creates a hidden .netrc file (_netrc for Windows OS) with Earthdata login credentials in your home directory. This file is needed to access NASA Earthdata assets from scripting environments like Python and R. Below is code describing this process.\n\nPythonR\n\n\nPlease see this 2021 Cloud Hackathon tutorial for Python code.\n\n\nFor authentication in R, please run the code below:\n\n\n\n[R code]\n\n# Required packages for this script\npackages = c('sys', 'getPass')\n\n# Identify missing (not installed) packages\nnew.packages = packages[!(packages %in% installed.packages()[,\"Package\"])]\n\n# Install missing packages\nif(length(new.packages)) install.packages(new.packages, repos='http://cran.rstudio.com/')\n\n# Load packages into R\nlibrary(sys)\nlibrary(getPass)\n\n# Determine OS and associated netrc file \nnetrc_type <- if(.Platform$OS.type == \"windows\") \"_netrc\" else \".netrc\" # Windows OS uses _netrc file\n\n# Specify path to user profile \nup <- file.path(Sys.getenv(\"USERPROFILE\")) # Retrieve user directory (for netrc file)\n\n# Below, HOME and Userprofile directories are set. \n\nif (up == \"\") {\n up <- Sys.getenv(\"HOME\") \n Sys.setenv(\"userprofile\" = up)\n if (up == \"\") {\n cat('USERPROFILE/HOME directories need to be set up. Please type sys.setenv(\"HOME\" = \"YOURDIRECTORY\") or sys.setenv(\"USERPROFILE\" = \"YOURDIRECTORY\") in your console and type your USERPROFILE/HOME direcory instead of \"YOURDIRECTORY\". Next, run the code chunk again.')\n }\n} else {Sys.setenv(\"HOME\" = up)} \n\nnetrc_path <- file.path(up, netrc_type, fsep = .Platform$file.sep) # Path to netrc file\n\n# Create a netrc file if one does not exist already\nif (file.exists(netrc_path) == FALSE || grepl(\"urs.earthdata.nasa.gov\", readLines(netrc_path)) == FALSE) {\n netrc_conn <- file(netrc_path)\n \n # User will be prompted for NASA Earthdata Login Username and Password below\n writeLines(c(\"machine urs.earthdata.nasa.gov\",\n sprintf(\"login %s\", getPass(msg = \"Enter NASA Earthdata Login Username \\n (An account can be Created at urs.earthdata.nasa.gov):\")),\n sprintf(\"password %s\", getPass(msg = \"Enter NASA Earthdata Login Password:\"))), netrc_conn)\n close(netrc_conn)\n}else{\n i <- 0 \n for (f in readLines(netrc_path)){\n i <- i + 1\n if (f ==\"machine urs.earthdata.nasa.gov\"){\n username <- strsplit(readLines(netrc_path)[i+1], \" \")[[1]][2]\n un <- getPass(msg = paste0(\"Is your NASA Earthdata Login Username: \", username, \"\\n\\n Type yes or no.\"))\n if (tolower(un) == 'yes'){\n tx <- gsub(readLines(netrc_path)[i+2], sprintf(\"password %s\", getPass(msg = \"Enter NASA Earthdata Login Password:\")), readLines(netrc_path))\n writeLines(tx, netrc_path)\n rm(username, un, tx, f, i)\n }else{\n user <- gsub(readLines(netrc_path)[i+1], sprintf(\"login %s\", getPass(msg = \"Enter NASA Earthdata Login Username:\")), readLines(netrc_path))\n tx <- gsub(readLines(netrc_path)[i+2], sprintf(\"password %s\", getPass(msg = \"Enter NASA Earthdata Login Password:\")), readLines(netrc_path))\n writeLines(tx, netrc_path)\n rm(username, un, user, tx, f, i)\n \n }\n break\n }\n }\n}" }, { - "objectID": "how-tos/access-data/access-cloud-python.html", - "href": "how-tos/access-data/access-cloud-python.html", - "title": "How do I access data stored in Earthdata Cloud in Python?", - "section": "", - "text": "When you have found the data you want to use, you have two options. You can download the data to work locally, or access the data directly to work in the cloud. This second way of working is called “Direct Cloud Access” or simply, “Direct Access”. Your compute instance needs to be in Amazon Web Services (AWS) Region us-west-2 in order for the code to run successfully. We authenticate using a netrc file and an Earthdata Login, see the appendix for more information on Earthdata Login and netrc setup." + "objectID": "appendix/authentication.html#aws-credentials", + "href": "appendix/authentication.html#aws-credentials", + "title": "Authentication", + "section": "AWS Credentials", + "text": "AWS Credentials\ncoming soon" }, { - "objectID": "how-tos/access-data/access-cloud-python.html#direct-access", - "href": "how-tos/access-data/access-cloud-python.html#direct-access", - "title": "How do I access data stored in Earthdata Cloud in Python?", + "objectID": "policies-usage/index.html", + "href": "policies-usage/index.html", + "title": "Policies & Usage Costs", "section": "", - "text": "When you have found the data you want to use, you have two options. You can download the data to work locally, or access the data directly to work in the cloud. This second way of working is called “Direct Cloud Access” or simply, “Direct Access”. Your compute instance needs to be in Amazon Web Services (AWS) Region us-west-2 in order for the code to run successfully. We authenticate using a netrc file and an Earthdata Login, see the appendix for more information on Earthdata Login and netrc setup." + "text": "Our current access policies are found here: https://github.com/NASA-Openscapes/2i2cAccessPolicies" }, { - "objectID": "how-tos/access-data/access-cloud-python.html#python", - "href": "how-tos/access-data/access-cloud-python.html#python", - "title": "How do I access data stored in Earthdata Cloud in Python?", - "section": "Python", - "text": "Python\nWe can use the earthaccess python library to grab the file URLs and then access them with the xarray library.\n\n#Import packages\nimport earthaccess\nimport xarray as xr\n\n\n#Authentication with Earthdata Login\nauth = earthaccess.login(strategy=\"netrc\")\n\n\n#Access land ice height from ATLAS/ICESat-2 V005 (10.5067/ATLAS/ATL06.005), searching for data over western Greenland coast over two weeks in July 2022. The data are provided as HDF5 granules (files) that span about 1/14th of an orbit.\n\nresults = earthaccess.search_data(short_name=\"ATL06\",\n version=\"006\",\n cloud_hosted=True,\n temporal = (\"2022-07-17\",\"2022-07-31\"),\n bounding_box = (-51.96423,68.10554,-48.71969,70.70529))\n\nGranules found: 5\n\n\n\n#Use xarray to load the data as a multifile dataset for a single group in the HDF5 file, in this case land ice segments:\nds = xr.open_mfdataset(earthaccess.open(results), group='/gt1l/land_ice_segments')\nds\n\n Opening 5 granules, approx size: 0.0 GB\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (delta_time: 241711)\nCoordinates:\n * delta_time (delta_time) datetime64[ns] 2022-07-18T01:00:46.67...\n latitude (delta_time) float64 dask.array<chunksize=(78325,), meta=np.ndarray>\n longitude (delta_time) float64 dask.array<chunksize=(78325,), meta=np.ndarray>\nData variables:\n atl06_quality_summary (delta_time) int8 dask.array<chunksize=(78325,), meta=np.ndarray>\n h_li (delta_time) float32 dask.array<chunksize=(78325,), meta=np.ndarray>\n h_li_sigma (delta_time) float32 dask.array<chunksize=(78325,), meta=np.ndarray>\n segment_id (delta_time) float64 dask.array<chunksize=(78325,), meta=np.ndarray>\n sigma_geo_h (delta_time) float32 dask.array<chunksize=(78325,), meta=np.ndarray>\nAttributes:\n Description: The land_ice_height group contains the primary set of deriv...\n data_rate: Data within this group are sparse. Data values are provide...xarray.DatasetDimensions:delta_time: 241711Coordinates: (3)delta_time(delta_time)datetime64[ns]2022-07-18T01:00:46.678760592 .....contentType :referenceInformationdescription :Number of GPS seconds since the ATLAS SDP epoch. The ATLAS Standard Data Products (SDP) epoch offset is defined within /ancillary_data/atlas_sdp_gps_epoch as the number of GPS seconds between the GPS epoch (1980-01-06T00:00:00.000000Z UTC) and the ATLAS SDP epoch. By adding the offset contained within atlas_sdp_gps_epoch to delta time parameters, the time in gps_seconds relative to the GPS epoch can be computed.long_name :Elapsed GPS secondssource :section 4.4standard_name :timearray(['2022-07-18T01:00:46.678760592', '2022-07-18T01:00:46.681322640',\n '2022-07-18T01:00:46.684008720', ..., '2022-07-26T00:49:18.826637808',\n '2022-07-26T00:49:18.829449568', '2022-07-26T00:49:18.832263232'],\n dtype='datetime64[ns]')latitude(delta_time)float64dask.array<chunksize=(78325,), meta=np.ndarray>contentType :physicalMeasurementdescription :Latitude of segment center, WGS84, North=+,long_name :Latitudesource :section 3.10standard_name :latitudeunits :degrees_northvalid_max :90.0valid_min :-90.0\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n1.84 MiB\n611.91 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat64 numpy.ndarray\n\n\n\n\n\n\n\n\nlongitude\n\n\n(delta_time)\n\n\nfloat64\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nphysicalMeasurement\n\ndescription :\n\nLongitude of segment center, , WGS84, East=+\n\nlong_name :\n\nLongitude\n\nsource :\n\nsection 3.10\n\nstandard_name :\n\nlongitude\n\nunits :\n\ndegrees_east\n\nvalid_max :\n\n180.0\n\nvalid_min :\n\n-180.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n1.84 MiB\n611.91 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat64 numpy.ndarray\n\n\n\n\n\n\n\n\nData variables: (5)\n\n\n\n\n\natl06_quality_summary\n\n\n(delta_time)\n\n\nint8\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nqualityInformation\n\ndescription :\n\nThe ATL06_quality_summary parameter indicates the best-quality subset of all ATL06 data. A zero in this parameter implies that no data-quality tests have found a problem with the segment, a one implies that some potential problem has been found. Users who select only segments with zero values for this flag can be relatively certain of obtaining high-quality data, but will likely miss a significant fraction of usable data, particularly in cloudy, rough, or low-surface-reflectance conditions.\n\nflag_meanings :\n\nbest_quality potential_problem\n\nflag_values :\n\n[0 1]\n\nlong_name :\n\nATL06_Quality_Summary\n\nsource :\n\nsection 4.3\n\nunits :\n\n1\n\nvalid_max :\n\n1\n\nvalid_min :\n\n0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n236.05 kiB\n76.49 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nint8 numpy.ndarray\n\n\n\n\n\n\n\n\n\nh_li\n\n\n(delta_time)\n\n\nfloat32\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nphysicalMeasurement\n\ndescription :\n\nStandard land-ice segment height determined by land ice algorithm, corrected for first-photon bias, representing the median- based height of the selected PEs\n\nlong_name :\n\nLand Ice height\n\nsource :\n\nsection 4.4\n\nunits :\n\nmeters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.92 MiB\n305.96 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nh_li_sigma\n\n\n(delta_time)\n\n\nfloat32\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nqualityInformation\n\ndescription :\n\nPropagated error due to sampling error and FPB correction from the land ice algorithm\n\nlong_name :\n\nExpected RMS segment misfit\n\nsource :\n\nsection 4.4\n\nunits :\n\nmeters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.92 MiB\n305.96 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nsegment_id\n\n\n(delta_time)\n\n\nfloat64\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nreferenceInformation\n\ndescription :\n\nSegment number, counting from the equator. Equal to the segment_id for the second of the two 20m ATL03 segments included in the 40m ATL06 segment\n\nlong_name :\n\nReference Point, m\n\nsource :\n\nsection 3.1.2.1\n\nunits :\n\n1\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n1.84 MiB\n611.91 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat64 numpy.ndarray\n\n\n\n\n\n\n\n\n\nsigma_geo_h\n\n\n(delta_time)\n\n\nfloat32\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nqualityInformation\n\ndescription :\n\nTotal vertical geolocation error due to PPD and POD, including the effects of horizontal geolocation error on the segment vertical error.\n\nlong_name :\n\nVertical Geolocation Error\n\nsource :\n\nATBD Section 3.10\n\nunits :\n\nmeters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.92 MiB\n305.96 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nIndexes: (1)delta_timePandasIndexPandasIndex(DatetimeIndex(['2022-07-18 01:00:46.678760592',\n '2022-07-18 01:00:46.681322640',\n '2022-07-18 01:00:46.684008720',\n '2022-07-18 01:00:46.686753504',\n '2022-07-18 01:00:46.689526560',\n '2022-07-18 01:00:46.692315280',\n '2022-07-18 01:00:46.695049040',\n '2022-07-18 01:00:46.700724096',\n '2022-07-18 01:00:46.703545872',\n '2022-07-18 01:00:46.706366832',\n ...\n '2022-07-26 00:49:18.806914512',\n '2022-07-26 00:49:18.809737328',\n '2022-07-26 00:49:18.812559600',\n '2022-07-26 00:49:18.815380608',\n '2022-07-26 00:49:18.818200224',\n '2022-07-26 00:49:18.821015744',\n '2022-07-26 00:49:18.823827088',\n '2022-07-26 00:49:18.826637808',\n '2022-07-26 00:49:18.829449568',\n '2022-07-26 00:49:18.832263232'],\n dtype='datetime64[ns]', name='delta_time', length=241711, freq=None))Attributes: (2)Description :The land_ice_height group contains the primary set of derived ATL06 products. This includes geolocation, height, and standard error and quality measures for each segment. This group is sparse, meaning that parameters are provided only for pairs of segments for which at least one beam has a valid surface-height measurement.data_rate :Data within this group are sparse. Data values are provided only for those ICESat-2 20m segments where at least one beam has a valid land ice height measurement." + "objectID": "policies-usage/index.html#nasa-openscapes-policies", + "href": "policies-usage/index.html#nasa-openscapes-policies", + "title": "Policies & Usage Costs", + "section": "", + "text": "Our current access policies are found here: https://github.com/NASA-Openscapes/2i2cAccessPolicies" }, { - "objectID": "how-tos/access-data/access-cloud-python.html#end-user-license-agreement-eula", - "href": "how-tos/access-data/access-cloud-python.html#end-user-license-agreement-eula", - "title": "How do I access data stored in Earthdata Cloud in Python?", - "section": "End User License Agreement (EULA)", - "text": "End User License Agreement (EULA)\nSometimes, accessing data in NASA Earthdata Cloud requires an End-User License Agreement (EULA). If you cannot access a dataset, this may be your issue! See these instructions for how to authorize EULAs." + "objectID": "policies-usage/index.html#aws-cost-explorer", + "href": "policies-usage/index.html#aws-cost-explorer", + "title": "Policies & Usage Costs", + "section": "AWS Cost Explorer", + "text": "AWS Cost Explorer\nThis is a work in progress, currently with minimal steps and screenshots that we will augment.\nAWS Cost Explorer lets you examine how much your usage costs. When using Credits, your usage does not immediately show up. Select Charge type as “Usage” from the right menu.\n\n\n\nAWS Cost Explorer. Charge type == “Usage”" }, { - "objectID": "how-tos/access-data/access-cloud-python.html#alternative-access-method-without-earthaccess", - "href": "how-tos/access-data/access-cloud-python.html#alternative-access-method-without-earthaccess", - "title": "How do I access data stored in Earthdata Cloud in Python?", - "section": "Alternative Access Method without earthaccess", - "text": "Alternative Access Method without earthaccess\nAn alternative approach to accessing data is outlined in some notebooks in the Earthdata Cloud Cookbook Appendix! The earthaccess package uses these methods for it’s back end. See this GitHub folder." + "objectID": "policies-usage/index.html#aws-budgeting-alerts", + "href": "policies-usage/index.html#aws-budgeting-alerts", + "title": "Policies & Usage Costs", + "section": "AWS Budgeting Alerts", + "text": "AWS Budgeting Alerts\nThis is a work in progress, currently with minimal steps and screenshots that we will augment.\nThere are two types of alerts we set up.\n\nBudgeting alerts\nWhen adding new Cloud credits to our AWS account, we also create a budget and alerts (received via email) as we spend our credits. These are some beginning notes (credit and thank you to Joe Kennedy!).\nCreate an annual budget of the total Credits left (you may need to calculate this if credits rolled over at the beginning of the calendar year). In the Budget menu, create a Budget. Then select Customize and Cost budget.\n\n\n\n\n\nExclude Credits and Refunds, include Discounts. You can elect to receive emails with a regular cadence: weekly, monthly.\nWe set these up at 50, 75, 90, 95% of the total budget, and we will receive emails at those percentages. The thinking is that we will need to request more credits starting at 50-75%, and then make sure we have them in hand by 90-95%.\n\n\nThreshold alerts\nWe can also set up email alerts at certain dollar amounts.\n\n\n\n\n\nWe receive emails when we spend $100, $200, $500 of our Credits, which show up in the system as $1000 intervals." }, { - "objectID": "how-tos/use_apis.html", - "href": "how-tos/use_apis.html", - "title": "How do I access data using APIs?", + "objectID": "tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html", + "href": "tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html", + "title": "Hurricanes: Wind and Sea Surface Temperature", "section": "", - "text": "API stands for Application Programming Interface, and is basically “like a digital mediator that enables different software applications to talk to each other and exchange information. It defines the methods, parameters, and data formats you can use to access certain features or retrieve data from an application or service.” (How to Use An API)" + "text": "This notebook demonstrates how to access two cloud-hosted Earthdata granules using the Commmon Metadata Repository (CMR) API. The granules are from two different DAACs (GES DISC and PO.DAAC). It shows the process of obtaining and plotting two variables from two distinct granules hosted in S3 buckets: sea surface temperature (SST) from the GHRSST Level 4 MUR Global Foundation Sea Surface Temperature Analysis (v4.1) (GHRSST)) and calculated 2-meter (2M) wind velocity from the MERRA-2 hourly time-averaged reanalysis dataset (M2T1NXSLV.5.12.4), before plotting them together on 25 October 2012 over the Caribbean Sea when Hurricane Sandy was at peak strength." }, { - "objectID": "how-tos/use_apis.html#using-apis", - "href": "how-tos/use_apis.html#using-apis", - "title": "How do I access data using APIs?", + "objectID": "tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html#overview", + "href": "tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html#overview", + "title": "Hurricanes: Wind and Sea Surface Temperature", "section": "", - "text": "API stands for Application Programming Interface, and is basically “like a digital mediator that enables different software applications to talk to each other and exchange information. It defines the methods, parameters, and data formats you can use to access certain features or retrieve data from an application or service.” (How to Use An API)" + "text": "This notebook demonstrates how to access two cloud-hosted Earthdata granules using the Commmon Metadata Repository (CMR) API. The granules are from two different DAACs (GES DISC and PO.DAAC). It shows the process of obtaining and plotting two variables from two distinct granules hosted in S3 buckets: sea surface temperature (SST) from the GHRSST Level 4 MUR Global Foundation Sea Surface Temperature Analysis (v4.1) (GHRSST)) and calculated 2-meter (2M) wind velocity from the MERRA-2 hourly time-averaged reanalysis dataset (M2T1NXSLV.5.12.4), before plotting them together on 25 October 2012 over the Caribbean Sea when Hurricane Sandy was at peak strength." }, { - "objectID": "how-tos/use_apis.html#how-do-i-know", - "href": "how-tos/use_apis.html#how-do-i-know", - "title": "How do I access data using APIs?", - "section": "How do I know?", - "text": "How do I know?\nHow do I know what API services are available for a dataset?\nSome but not all datasets have API services that can reproject, resample or reformat files on the server file. These How-To’s below show how to find which services are available:\n\nFinding Services using EarthData Search\nAfter selecting a dataset and the needed files using Earthdata Search (tutorial here), when the download files button is selected, it will navigate users to a new page on the sidebar. If the dataset is customizable through APIs, different options will pop up under “Select a data access method” or “Configure data customization options”\n\n\nFinding Services using CMR\nFollow this link to see how to programmatically discover what services are associated with what NASA Earthdata." + "objectID": "tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html#learning-objectives", + "href": "tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html#learning-objectives", + "title": "Hurricanes: Wind and Sea Surface Temperature", + "section": "Learning Objectives", + "text": "Learning Objectives\n\nSearch for data programmatically using known parameters\nAccess data within the AWS cloud using the earthaccess python library\nAccess data archived by 2 NASA data centers from the AWS cloud\nVisualize wind speed and sea surface temperature together during a hurricane event" }, { - "objectID": "how-tos/use_apis.html#harmony", - "href": "how-tos/use_apis.html#harmony", - "title": "How do I access data using APIs?", - "section": "Harmony", - "text": "Harmony\nHarmony allows you to seamlessly analyze Earth observation data from different NASA data centers with services that offer data reduction and common access mechanisms across all NASA data centers. There is a python client library called harmony-py that exists as a wrapper for the Harmony services, and this notebook gives a brief introduction to the Harmony API services." + "objectID": "tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html#requirements", + "href": "tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html#requirements", + "title": "Hurricanes: Wind and Sea Surface Temperature", + "section": "Requirements", + "text": "Requirements\n\n1. Compute environment - This notebook can only be run in an AWS Cloud instance running in the us-west-2 region.\n\n\n2. Earthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\nInstall Packages\n\nimport earthaccess\nimport xarray as xr\nimport requests\nimport s3fs\nimport numpy as np\nimport pprint\nimport cartopy.crs as ccrs\nimport cartopy.feature as cfeature\nimport matplotlib.pyplot as plt\nimport boto3\nfrom IPython.display import display, Markdown\n\n%matplotlib inline\n\nWe recommend authenticating your Earthdata Login (EDL) information using the earthaccess python package as follows:\n\nauth = earthaccess.login(strategy=\"netrc\") # works if the EDL login already been persisted to a netrc\nif not auth.authenticated:\n # ask for EDL credentials and persist them in a .netrc file\n auth = earthaccess.login(strategy=\"interactive\", persist=True)\n\nYou're now authenticated with NASA Earthdata Login\nearthaccess generated a token for CMR with expiration on: 10/29/2023\nearthaccess generated a token for CMR with expiration on: 10/29/2023\nUsing .netrc file for EDL\n\n\n\n\nRetrieve Granule S3 Links\nTo retrieve the S3 URLs, we will use the CMR Search API to pass the “Short Name” of the dataset, the version of the dataset, the temporal range, and the page size for the JSON response. For GES DISC datasets, you can find the short name and version by navigating to any dataset landing page, where it will be listed in the “Product Summary” tab. We will also retrieve the S3 token endpoints, used for accessing the S3 buckets with our Earthdata credentials:\n\ncmr_url = 'https://cmr.earthdata.nasa.gov/search/granules'\n\nmerra_response = requests.get(cmr_url, \n params={\n 'short_name': 'M2T1NXSLV',\n 'version': '5.12.4',\n 'temporal': '2012-10-25T00:00:00Z',\n 'page_size': 1,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\n\n# Parse S3 Token Endpoint and S3 URL\n\nprint('S3 Token Endpoint:')\n\nfor element in merra_response.json()['feed']['entry'][0]['links']:\n if element['href'].endswith('s3credentials'):\n print(element['href'])\n gesdisc_s3 = element['href']\n \nprint('S3 URL:')\n\nfor element in merra_response.json()['feed']['entry'][0]['links']:\n if element['rel'] == 'http://esipfed.org/ns/fedsearch/1.1/s3#':\n print(element['href'])\n merra_fn = element['href']\n\nS3 Token Endpoint:\nhttps://data.gesdisc.earthdata.nasa.gov/s3credentials\nS3 URL:\ns3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2012/10/MERRA2_400.tavg1_2d_slv_Nx.20121025.nc4\n\n\nThen, we build a similar request for retrieving our sea surface temperature data from the GHRSST dataset at 2012-10-25 00Z. The short name and version for this dataset was retrieved from its PO.DAAC dataset landing page, under the “Information” tab.\n\nghrsst_response = requests.get(cmr_url, \n params={\n 'short_name': 'MUR-JPL-L4-GLOB-v4.1',\n 'version': '4.1',\n 'temporal': '2012-10-25T00:00:00Z',\n 'page_size': 1,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\n\n# Parse S3 Token Endpoint and S3 URL\n\nprint('S3 Token Endpoint:')\n\nfor element in ghrsst_response.json()['feed']['entry'][0]['links']:\n if element['href'].endswith('s3credentials'):\n print(element['href'])\n podaac_s3 = element['href']\n \nprint('S3 URL:')\n\nfor element in ghrsst_response.json()['feed']['entry'][0]['links']:\n if element['rel'] == 'http://esipfed.org/ns/fedsearch/1.1/s3#':\n print(element['href'])\n ghrsst_fn = element['href']\n\nS3 Token Endpoint:\nhttps://archive.podaac.earthdata.nasa.gov/s3credentials\nS3 URL:\ns3://podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20121025090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc\n\n\n\n\nObtain S3 Credentials and Open Bucket Granules\nDirect S3 access is granted to the user through a temporary token, which will last for one hour, and will need to be rerun after that hour has passed to access the bucket. Then, to access the bucket, we can pass this token into the S3FS S3FileSystem() function which will “mount” the S3 bucket to our notebook as if it were a locally stored file system. Here, we create a function that queries the S3 Token Endpoint with their Earthdata credentials, stored in a netrc file, using the Python requests library. Next, it returns an s3fs.core.S3FileSystem object that represents the “mounted” S3 bucket:\n\n# Define a function for S3 access credentials\n\ndef begin_s3_direct_access(daac_url):\n \n # Retrieve the token as a JSON\n response = requests.get(daac_url).json()\n \n # Mount the bucket and return it as an S3FileSystem object\n return s3fs.S3FileSystem(key=response['accessKeyId'],\n secret=response['secretAccessKey'],\n token=response['sessionToken'],\n client_kwargs={'region_name':'us-west-2'})\n\nWe need two tokens in order to access the two different buckets, so we will perform two function calls with the two S3 Token Endpoints retrieved from CMR, stored in separate variables. If you immediately receive an error, double-check that your username and password were entered correctly in your netrc file, or that you can access the following URL: https://data.gesdisc.earthdata.nasa.gov\n\n# Open S3 file systems with S3FS\n\ngesdisc_fs = begin_s3_direct_access(gesdisc_s3)\npodaac_fs = begin_s3_direct_access(podaac_s3)\n\n# Check that the file system is intact as an S3FileSystem object, which means that token is valid\n\ntype(gesdisc_fs)\ntype(podaac_fs)\n\ns3fs.core.S3FileSystem\n\n\n\n\nOpen Granules in Xarray\nIn order to open the granules in Xarray, we must use the S3FS open() function. Here, we pass the S3 URLs from each separate file system into two different variables:\n\n# Open datasets with S3FS\n\nmerra_ds = xr.open_dataset(gesdisc_fs.open(merra_fn))\nghrsst_ds = xr.open_dataset(podaac_fs.open(ghrsst_fn))\n\nNow, the granules are stored in memory as Xarray datasets, which will be stored as long as the kernel is running. Here, we clip to the extent over where the hurricane was:\n\nmin_lon = -89\nmin_lat = 14\nmax_lon = -67\nmax_lat = 31\n\nmerra_ds = merra_ds.sel(lat=slice(min_lat,max_lat), lon=slice(min_lon,max_lon))\nghrsst_ds = ghrsst_ds.sel(lat=slice(min_lat,max_lat), lon=slice(min_lon,max_lon))\n\n\n\nConvert Dataset Grids\nHere, we interpolate the GHRSST grid to the MERRA grid using Xarray’s interp() function:\n\nghrsst_ds = ghrsst_ds.interp(lat=merra_ds.lat, lon=merra_ds.lon)\nghrsst_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (time: 1, lat: 35, lon: 35)\nCoordinates:\n * time (time) datetime64[ns] 2012-10-25T09:00:00\n * lat (lat) float64 14.0 14.5 15.0 15.5 ... 29.5 30.0 30.5 31.0\n * lon (lon) float64 -88.75 -88.12 -87.5 ... -68.75 -68.12 -67.5\nData variables:\n analysed_sst (time, lat, lon) float64 nan nan nan ... 300.0 299.4 299.4\n analysis_error (time, lat, lon) float64 nan nan nan ... 0.37 0.37 0.37\n mask (time, lat, lon) float64 2.0 2.0 2.0 2.0 ... 1.0 1.0 1.0\n sea_ice_fraction (time, lat, lon) float64 nan nan nan nan ... nan nan nan\nAttributes: (12/47)\n Conventions: CF-1.5\n title: Daily MUR SST, Final product\n summary: A merged, multi-sensor L4 Foundation SST anal...\n references: http://podaac.jpl.nasa.gov/Multi-scale_Ultra-...\n institution: Jet Propulsion Laboratory\n history: created at nominal 4-day latency; replaced nr...\n ... ...\n project: NASA Making Earth Science Data Records for Us...\n publisher_name: GHRSST Project Office\n publisher_url: http://www.ghrsst.org\n publisher_email: ghrsst-po@nceo.ac.uk\n processing_level: L4\n cdm_data_type: gridxarray.DatasetDimensions:time: 1lat: 35lon: 35Coordinates: (3)time(time)datetime64[ns]2012-10-25T09:00:00long_name :reference time of sst fieldstandard_name :timeaxis :Tcomment :Nominal time of analyzed fieldsarray(['2012-10-25T09:00:00.000000000'], dtype='datetime64[ns]')lat(lat)float6414.0 14.5 15.0 ... 30.0 30.5 31.0long_name :latitudeunits :degrees_northvmax :1000000000000000.0vmin :-1000000000000000.0valid_range :[-1.e+15 1.e+15]array([14. , 14.5, 15. , 15.5, 16. , 16.5, 17. , 17.5, 18. , 18.5, 19. , 19.5,\n 20. , 20.5, 21. , 21.5, 22. , 22.5, 23. , 23.5, 24. , 24.5, 25. , 25.5,\n 26. , 26.5, 27. , 27.5, 28. , 28.5, 29. , 29.5, 30. , 30.5, 31. ])lon(lon)float64-88.75 -88.12 ... -68.12 -67.5long_name :longitudeunits :degrees_eastvmax :1000000000000000.0vmin :-1000000000000000.0valid_range :[-1.e+15 1.e+15]array([-88.75 , -88.125, -87.5 , -86.875, -86.25 , -85.625, -85. , -84.375,\n -83.75 , -83.125, -82.5 , -81.875, -81.25 , -80.625, -80. , -79.375,\n -78.75 , -78.125, -77.5 , -76.875, -76.25 , -75.625, -75. , -74.375,\n -73.75 , -73.125, -72.5 , -71.875, -71.25 , -70.625, -70. , -69.375,\n -68.75 , -68.125, -67.5 ])Data variables: (4)analysed_sst(time, lat, lon)float64nan nan nan ... 300.0 299.4 299.4long_name :analysed sea surface temperaturestandard_name :sea_surface_foundation_temperatureunits :kelvinvalid_min :-32767valid_max :32767comment :\"Final\" version using Multi-Resolution Variational Analysis (MRVA) method for interpolationsource :AVHRR18_G-NAVO, AVHRR19_G-NAVO, AVHRR_METOP_A-EUMETSAT, MODIS_A-JPL, MODIS_T-JPL, WSAT-REMSS, iQUAM-NOAA/NESDIS, Ice_Conc-OSISAFarray([[[ nan, nan, nan, ..., 302.08099365,\n 301.34849548, 302.09698486],\n [ nan, nan, nan, ..., 301.13699341,\n 301.77400208, 301.94799805],\n [ nan, nan, nan, ..., 302.04498291,\n 302.22299194, 302.26998901],\n ...,\n [297.48300171, 298.64399719, 298.88000488, ..., 300.36499023,\n 300.13198853, 299.65600586],\n [ nan, nan, nan, ..., 300.27600098,\n 299.01649475, 299.55300903],\n [ nan, nan, nan, ..., 300.01699829,\n 299.39898682, 299.44500733]]])analysis_error(time, lat, lon)float64nan nan nan nan ... 0.37 0.37 0.37long_name :estimated error standard deviation of analysed_sstunits :kelvinvalid_min :0valid_max :32767comment :nonearray([[[ nan, nan, nan, ..., 0.38 ,\n 0.38 , 0.38 ],\n [ nan, nan, nan, ..., 0.38 ,\n 0.38 , 0.38999999],\n [ nan, nan, nan, ..., 0.38999999,\n 0.38 , 0.39999998],\n ...,\n [0.37 , 0.37 , 0.37 , ..., 0.37 ,\n 0.37 , 0.37 ],\n [ nan, nan, nan, ..., 0.37 ,\n 0.38999999, 0.37 ],\n [ nan, nan, nan, ..., 0.37 ,\n 0.37 , 0.37 ]]])mask(time, lat, lon)float642.0 2.0 2.0 2.0 ... 1.0 1.0 1.0 1.0long_name :sea/land field composite maskvalid_min :1valid_max :31flag_masks :[ 1 2 4 8 16]flag_values :[ 1 2 5 9 13]flag_meanings :1=open-sea, 2=land, 5=open-lake, 9=open-sea with ice in the grid, 13=open-lake with ice in the gridcomment :mask can be used to further filter the data.source :GMT \"grdlandmask\", ice flag from sea_ice_fraction dataarray([[[2., 2., 2., ..., 1., 1., 1.],\n [2., 2., 2., ..., 1., 1., 1.],\n [2., 2., 2., ..., 1., 1., 1.],\n ...,\n [1., 1., 1., ..., 1., 1., 1.],\n [2., 2., 2., ..., 1., 1., 1.],\n [2., 2., 2., ..., 1., 1., 1.]]])sea_ice_fraction(time, lat, lon)float64nan nan nan nan ... nan nan nan nanlong_name :sea ice area fractionstandard_name :sea ice area fractionunits :fraction (between 0 and 1)valid_min :0valid_max :100source :EUMETSAT OSI-SAF, copyright EUMETSATcomment :ice data interpolated by a nearest neighbor approach.array([[[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]]])Indexes: (3)timePandasIndexPandasIndex(DatetimeIndex(['2012-10-25 09:00:00'], dtype='datetime64[ns]', name='time', freq=None))latPandasIndexPandasIndex(Index([14.0, 14.5, 15.0, 15.5, 16.0, 16.5, 17.0, 17.5, 18.0, 18.5, 19.0, 19.5,\n 20.0, 20.5, 21.0, 21.5, 22.0, 22.5, 23.0, 23.5, 24.0, 24.5, 25.0, 25.5,\n 26.0, 26.5, 27.0, 27.5, 28.0, 28.5, 29.0, 29.5, 30.0, 30.5, 31.0],\n dtype='float64', name='lat'))lonPandasIndexPandasIndex(Index([ -88.75, -88.125, -87.5, -86.875, -86.25, -85.625, -85.0, -84.375,\n -83.75, -83.125, -82.5, -81.875, -81.25, -80.625, -80.0, -79.375,\n -78.75, -78.125, -77.5, -76.875, -76.25, -75.625, -75.0, -74.375,\n -73.75, -73.125, -72.5, -71.875, -71.25, -70.625, -70.0, -69.375,\n -68.75, -68.125, -67.5],\n dtype='float64', name='lon'))Attributes: (47)Conventions :CF-1.5title :Daily MUR SST, Final productsummary :A merged, multi-sensor L4 Foundation SST analysis product from JPL.references :http://podaac.jpl.nasa.gov/Multi-scale_Ultra-high_Resolution_MUR-SSTinstitution :Jet Propulsion Laboratoryhistory :created at nominal 4-day latency; replaced nrt (1-day latency) version.comment :MUR = \"Multi-scale Ultra-high Reolution\"license :These data are available free of charge under data policy of JPL PO.DAAC.id :MUR-JPL-L4-GLOB-v04.1naming_authority :org.ghrsstproduct_version :04.1uuid :27665bc0-d5fc-11e1-9b23-0800200c9a66gds_version_id :2.0netcdf_version_id :4.1date_created :20150710T184006Zstart_time :20121025T090000Zstop_time :20121025T090000Ztime_coverage_start :20121024T210000Ztime_coverage_end :20121025T210000Zfile_quality_level :1source :AVHRR18_G-NAVO, AVHRR19_G-NAVO, AVHRR_METOP_A-EUMETSAT, MODIS_A-JPL, MODIS_T-JPL, WSAT-REMSS, iQUAM-NOAA/NESDIS, Ice_Conc-OSISAFplatform :Aqua, DMSP, NOAA-POES, Suomi-NPP, Terrasensor :AMSR-E, AVHRR, MODIS, SSM/I, VIIRS, in-situMetadata_Conventions :Unidata Observation Dataset v1.0metadata_link :http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MUR-JPL-L4-GLOB-v04.1keywords :Oceans > Ocean Temperature > Sea Surface Temperaturekeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordsstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventionsouthernmost_latitude :-90.0northernmost_latitude :90.0westernmost_longitude :-180.0easternmost_longitude :180.0spatial_resolution :0.01 degreesgeospatial_lat_units :degrees northgeospatial_lat_resolution :0.01 degreesgeospatial_lon_units :degrees eastgeospatial_lon_resolution :0.01 degreesacknowledgment :Please acknowledge the use of these data with the following statement: These data were provided by JPL under support by NASA MEaSUREs program.creator_name :JPL MUR SST projectcreator_email :ghrsst@podaac.jpl.nasa.govcreator_url :http://mur.jpl.nasa.govproject :NASA Making Earth Science Data Records for Use in Research Environments (MEaSUREs) Programpublisher_name :GHRSST Project Officepublisher_url :http://www.ghrsst.orgpublisher_email :ghrsst-po@nceo.ac.ukprocessing_level :L4cdm_data_type :grid\n\n\n\n\nPlot Variables using Matplotlib and Cartopy\nFinally, we use Matplotlib and Cartopy to plot a color mesh and quiver plot of SSTs and 2M winds of Hurricane Sandy at her peak strength:\n\n# Pre-configure wind vector variables\n\nu=merra_ds.U2M\nv=merra_ds.V2M\nlon=merra_ds.lon\nlat=merra_ds.lat\n\nlons, lats = np.meshgrid(lon, lat)\n\n# Plotting routines:\n\n# Figure size\nplt.rcParams['figure.figsize'] = 15, 15\n\n# Figure and geography setup\nfig = plt.figure()\nax = fig.add_subplot(2, 1, 1, projection=ccrs.PlateCarree())\nax.set_extent([-89, -67, 14, 31], crs=ccrs.PlateCarree())\nax.add_feature(cfeature.COASTLINE.with_scale('50m'), linewidth=0.5, zorder=5) \nax.add_feature(cfeature.LAND, facecolor='white', zorder=2) \nax.add_feature(cfeature.BORDERS, linewidth=0.5, zorder=5)\nax.add_feature(cfeature.STATES, zorder=5)\n\n# Colormesh of SSTs\nmmp = ax.pcolormesh(lons, lats, ghrsst_ds.analysed_sst.isel(time=0), \n cmap='hot_r', transform=ccrs.PlateCarree(), zorder=1)\n\n# Quiver plot of 2M vector field\nq = ax.quiver(lons, lats, u.isel(time=0).values, v.isel(time=0).values, zorder=4, \n transform=ccrs.PlateCarree(), scale_units='inches', color='gray')\n\n# Quiver key for scale\nax.quiverkey(q, 1.22, 1.05, 10, r'$10 \\frac{m}{s}$', zorder=4)\n\n# Lat/lon grid lines\nax.gridlines(draw_labels=True, dms=True, x_inline=False, y_inline=False)\n\n# SST color bar setup\ncbar = plt.colorbar(mmp, pad=0.1)\ncbar.set_label(\"Analyzed SST (K)\")\n\n# Figure title\nfig.suptitle(\"GHRSST Analyzed SST and MERRA-2 2M Wind Vectors on 2012-10-25T00:00:00Z\", size=16, y=0.95)\n\nText(0.5, 0.95, 'GHRSST Analyzed SST and MERRA-2 2M Wind Vectors on 2012-10-25T00:00:00Z')\n\n\n/srv/conda/envs/notebook/lib/python3.9/site-packages/cartopy/io/__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/50m_physical/ne_50m_land.zip\n warnings.warn(f'Downloading: {url}', DownloadWarning)\n/srv/conda/envs/notebook/lib/python3.9/site-packages/cartopy/io/__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/50m_cultural/ne_50m_admin_1_states_provinces_lakes.zip\n warnings.warn(f'Downloading: {url}', DownloadWarning)" }, { - "objectID": "how-tos/use_apis.html#opendap", - "href": "how-tos/use_apis.html#opendap", - "title": "How do I access data using APIs?", - "section": "OPeNDAP", - "text": "OPeNDAP\nOPeNDAP, or the Open-source Project for a Network Data Access Protocol, is a data server that allows for accessing scientific datasets through the public internet. OPeNDAP links can be acquired through dataset landing pages or the common metadata repository. For more information, see the OPeNDAP Data Access Client Tutorials." + "objectID": "tutorials/Data_stories_template.html", + "href": "tutorials/Data_stories_template.html", + "title": "Title", + "section": "", + "text": "Add Summary of data story/background information needed. 2-5 sentences." }, { - "objectID": "how-tos/use_apis.html#earthaccess", - "href": "how-tos/use_apis.html#earthaccess", - "title": "How do I access data using APIs?", - "section": "earthaccess", - "text": "earthaccess\nearthaccess is a python library to search, download or stream NASA Earth science data with just a few lines of code that also abstracts a lot of other APIs developed by NASA." + "objectID": "tutorials/Data_stories_template.html#overview", + "href": "tutorials/Data_stories_template.html#overview", + "title": "Title", + "section": "", + "text": "Add Summary of data story/background information needed. 2-5 sentences." }, { - "objectID": "how-tos/use_apis.html#appeears", - "href": "how-tos/use_apis.html#appeears", - "title": "How do I access data using APIs?", - "section": "AppEEARS", - "text": "AppEEARS\nThe Application for Extracting and Exploring Analysis Ready Samples (AρρEEARS) offers users a simple and efficient way to perform data access and transformation processes. By enabling users to subset data spatially, temporally, and by layer, the volume of data downloaded for analysis is greatly reduced. AppEEARS requests can be submitted in website and via API. In a same way, outputs generated can also be accessed from AppEEARS website and through API. View AppEEARS documentation and API documentation for more information. Additionally, you can work with the data in the cloud using provided S3 links. Visit AppEEARS Data Resources Repository for more tutorials on how to work with AppEEARS programaticaly." + "objectID": "tutorials/Data_stories_template.html#learning-objectives", + "href": "tutorials/Data_stories_template.html#learning-objectives", + "title": "Title", + "section": "Learning Objectives", + "text": "Learning Objectives\n\nenter objective\nenter objective\n…" }, { - "objectID": "how-tos/find-data/find-python.html", - "href": "how-tos/find-data/find-python.html", - "title": "How do I find data using Python?", - "section": "", - "text": "Here are our recommended approaches for finding data with Python, from the command line or a notebook.\nIn Python we can use the earthaccess library.\nThis example searches for data from the Land Processes DAAC with a spatial bounding box and temporal range.\n\n## Import packages\nimport earthaccess\nfrom earthaccess import DataGranules, DataCollections\nfrom pprint import pprint \n\n\n## We'll get 4 collections that match with our keyword of interest\ncollections = DataCollections().keyword(\"REFLECTANCE\").cloud_hosted(True).get(4)\n\n\n## Let's print 2 collections\nfor collection in collections[0:2]:\n print(pprint(collection.summary()) , collection.abstract(), \"\\n\")\n\n{'cloud-info': {'Region': 'us-west-2',\n 'S3BucketAndObjectPrefixNames': ['s3://lp-prod-protected/MOD09GA.061',\n 's3://lp-prod-public/MOD09GA.061'],\n 'S3CredentialsAPIDocumentationURL': 'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentialsREADME',\n 'S3CredentialsAPIEndpoint': 'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials'},\n 'concept-id': 'C2202497474-LPCLOUD',\n 'file-type': \"[{'Format': 'HDF-EOS2', 'FormatType': 'Native', 'Media': \"\n \"['HTTPS', 'Earthdata Cloud']}]\",\n 'get-data': ['https://e4ftl01.cr.usgs.gov/MOLT/MOD09GA.061/',\n 'https://search.earthdata.nasa.gov/search/granules?p=C2202497474-LPCLOUD',\n 'https://appeears.earthdatacloud.nasa.gov/',\n 'https://earthexplorer.usgs.gov/'],\n 'short-name': 'MOD09GA',\n 'version': '061'}\nNone The MOD09GA Version 6.1 product provides an estimate of the surface spectral reflectance of Terra Moderate Resolution Imaging Spectroradiometer (MODIS) Bands 1 through 7, corrected for atmospheric conditions such as gasses, aerosols, and Rayleigh scattering. Provided along with the 500 meter (m) surface reflectance, observation, and quality bands are a set of ten 1 kilometer (km) observation bands and geolocation flags. The reflectance layers from the MOD09GA are used as the source data for many of the MODIS land products. \n\nValidation at stage 3 (https://modis-land.gsfc.nasa.gov/MODLAND_val.html) has been achieved for the MODIS Surface Reflectance products. Further details regarding MODIS land product validation for the MOD09 data product is available from the MODIS Land Team Validation site (https://modis-land.gsfc.nasa.gov/ValStatus.php?ProductID=MOD09).\n\n Improvements/Changes from Previous Versions\n\n* The Version 6.1 Level-1B (L1B) products have been improved by undergoing various calibration changes that include: changes to the response-versus-scan angle (RVS) approach that affects reflectance bands for Aqua and Terra MODIS, corrections to adjust for the optical crosstalk in Terra MODIS infrared (IR) bands, and corrections to the Terra MODIS forward look-up table (LUT) update for the period 2012 - 2017.\n* A polarization correction has been applied to the L1B Reflective Solar Bands (RSB).\n\n \n\n{'cloud-info': {'Region': 'us-west-2',\n 'S3BucketAndObjectPrefixNames': ['s3://lp-prod-protected/MOD09GQ.061',\n 's3://lp-prod-public/MOD09GQ.061'],\n 'S3CredentialsAPIDocumentationURL': 'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentialsREADME',\n 'S3CredentialsAPIEndpoint': 'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials'},\n 'concept-id': 'C2343115666-LPCLOUD',\n 'file-type': \"[{'Format': 'HDF-EOS2', 'FormatType': 'Native', 'Media': \"\n \"['HTTPS', 'Earthdata Cloud']}]\",\n 'get-data': ['https://e4ftl01.cr.usgs.gov/MOLT/MOD09GQ.061/',\n 'https://search.earthdata.nasa.gov/search?q=C2343115666-LPCLOUD',\n 'https://earthexplorer.usgs.gov/',\n 'https://appeears.earthdatacloud.nasa.gov/'],\n 'short-name': 'MOD09GQ',\n 'version': '061'}\nNone The MOD09GQ Version 6.1 product provides an estimate of the surface spectral reflectance of Terra Moderate Resolution Imaging Spectroradiometer (MODIS) 250 meter (m) bands 1 and 2, corrected for atmospheric conditions such as gasses, aerosols, and Rayleigh scattering. Along with the 250 m surface reflectance bands are the Quality Assurance (QA) layer and five observation layers. This product is intended to be used in conjunction with the quality and viewing geometry information of the 500 m product (MOD09GA). \n\nValidation at stage 3 (https://modis-land.gsfc.nasa.gov/MODLAND_val.html) has been achieved for the MODIS Surface Reflectance products. Further details regarding MODIS land product validation for the MOD09 data product is available from the MODIS Land Team Validation site (https://modis-land.gsfc.nasa.gov/ValStatus.php?ProductID=MOD09).\n\nImprovements/Changes from Previous Versions\n\n* The Version 6.1 Level-1B (L1B) products have been improved by undergoing various calibration changes that include: changes to the response-versus-scan angle (RVS) approach that affects reflectance bands for Aqua and Terra MODIS, corrections to adjust for the optical crosstalk in Terra MODIS infrared (IR) bands, and corrections to the Terra MODIS forward look-up table (LUT) update for the period 2012 - 2017.\n* A polarization correction has been applied to the L1B Reflective Solar Bands (RSB). \n\n\n\n\n## Search for files from the second dataset result over a small plot in Nebraska, USA for two weeks in September 2022\ngranules = earthaccess.search_data(concept_id = \"C2021957657-LPCLOUD\", \n temporal = (\"2022-09-10\",\"2022-09-24\"),\n bounding_box = (-101.67271,41.04754,-101.65344,41.06213)) \n\nGranules found: 8" + "objectID": "tutorials/Data_stories_template.html#requirements", + "href": "tutorials/Data_stories_template.html#requirements", + "title": "Title", + "section": "Requirements", + "text": "Requirements\n\n1. Compute environment - This notebook can only be run in an AWS Cloud instance running in the us-west-2 region.\n\n\n2. Earthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up. ### 3. Additional Requirements …\n\n\nInstall Packages\n\nimport earthaccess\nimport xarray as xr\n\nWe recommend authenticating your Earthdata Login (EDL) information using the earthaccess python package as follows:\n\nauth = earthaccess.login(strategy=\"netrc\") # works if the EDL login already been persisted to a netrc\nif not auth.authenticated:\n # ask for EDL credentials and persist them in a .netrc file\n auth = earthaccess.login(strategy=\"interactive\", persist=True)" }, { - "objectID": "how-tos/find-data/earthdata_search.html", - "href": "how-tos/find-data/earthdata_search.html", - "title": "How do I find data using Earthdata Search?", + "objectID": "tutorials/index.html", + "href": "tutorials/index.html", + "title": "Tutorials", "section": "", - "text": "The original source for this document is https://nasa-openscapes.github.io/2021-Cloud-Workshop-AGU/tutorials/01_Earthdata_Search.html\nThis tutorial guides you through how to use Earthdata Search for NASA Earth observations search and discovery, and how to connect the search output (e.g. download or access links) to a programmatic workflow (locally or from within the cloud).\n\nStep 1. Go to Earthdata Search and Login\nGo to Earthdata Search https://search.earthdata.nasa.gov and use your Earthdata login credentials to log in. If you do not have an Earthdata account, please see the Workshop Prerequisites for guidance.\n\n\nStep 2. Search for dataset of interest\nUse the search box in the upper left to type key words. In this example we are interested in the ECCO dataset, hosted by the PO.DAAC. This dataset is available from the NASA Earthdata Cloud archive hosted in AWS cloud.\nClick on the “Available from AWS Cloud” filter option on the left. Here, 104 matching collections were found with the basic ECCO search.\n\n\n\nFigure caption: Search for ECCO data available in AWS cloud in Earthdata Search portal\n\n\nLet’s refine our search further. Let’s search for ECCO monthly SSH in the search box (which will produce 39 matching collections), and for the time period for year 2015. The latter can be done using the calendar icon on the left under the search box.\nScroll down the list of returned matches until we see the dataset of interest, in this case ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4).\nWe can click on the (i) icon for the dataset to read more details, including the dataset shortname (helpful for programmatic workflows) just below the dataset name; here ECCO_L4_SSH_05DEG_MONTHLY_V4R4.\n\n\n\nFigure caption: Refine search, set temporal bounds, get more information\n\n\n\n\nStep 3. Explore the dataset details, including Cloud Access information\nOnce we clicked the (i), scrolling down the info page for the dataset we will see Cloud Access information, such as:\n\nwhether the dataset is available in the cloud\nthe cloud Region (all NASA Earthdata Cloud data is/will be in us-west-2 region)\nthe S3 storage bucket and object prefix where this data is located\nlink that generates AWS S3 Credentials for in-cloud data access (we will cover this in the Direct Data Access Tutorials)\nlink to documentation describing the In-region Direct S3 Access to Buckets. Note: these will be unique depending on the DAAC where the data is archived. (We will show examples of direct in-region access in Tutorial 3.)\n\n\n\n\nFigure caption: Cloud access info in EDS\n\n\n\n\n\nFigure caption: Documentation describing the In-region Direct S3 Access to Buckets\n\n\nPro Tip: Clicking on “For Developers” to exapnd will provide programmatic endpoints such as those for the CMR API, and more. CMR API and CMR STAC API tutorials can be found on the 2021 Cloud Hackathon website.\nFor now, let’s say we are intersted in getting download link(s) or access link(s) for specific data files (granules) within this collection.\nAt the top of the dataset info section, click on Search Results, which will take us back to the list of datasets matching our search parameters. Clicking on the dataset (here again it’s the same ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4)) we now see a list of files (granules) that are part of the dataset (collection).\n\n\nStep 4. Customize the download or data access\nClick on the green + symbol to add a few files to our project. Here we added the first 3 listed for 2015. Then click on the green button towards the bottom that says “Download”. This will take us to another page with options to customize our download or access link(s).\n\n\n\nFigure caption: Select granules and click download\n\n\n\n4.a. Entire file content\nLet’s stay we are interested in the entire file content, so we select the “Direct Download” option (as opposed to other options to subset or transform the data):\n\n\n\nFigure caption: Customize your download or access\n\n\nClicking the green Download Data button again, will take us to the final page for instructions to download and links for data access in the cloud. You should see three tabs: Download Files, AWS S3 Access, Download Script:\n \nThe Download Files tab provides the https:// links for downloading the files locally. E.g.: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-09_ECCO_V4r4_latlon_0p50deg.nc\nThe AWS S3 Access tab provides the S3:// links, which is what we would use to access the data directly in-region (us-west-2) within the AWS cloud (an example will be shown in Tutorial 3). E.g.: s3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-09_ECCO_V4r4_latlon_0p50deg.nc where s3 indicates data is stored in AWS S3 storage, podaac-ops-cumulus-protected is the bucket, and ECCO_L4_SSH_05DEG_MONTHLY_V4R4 is the object prefix (the latter two are also listed in the dataset collection information under Cloud Access (step 3 above)).\nTip: Another quicker way to find the bucket and object prefix is from the list of data files the search returns. Next to the + green button is a grey donwload symbol. Click on that to see the Download Files https:// links or on the AWS S3 Access to get the direct S3:// access links, which contain the bucket and object prefix where data is stored.\n\n\n4.b. Subset or transform before download or access\nDAAC tools and services are also being migrated or developed in the cloud, next to that data. These include the Harmony API and OPeNDAP in the cloud, as a few examples.\nWe can leverage these cloud-based services on cloud-archived data to reduce or transform the data (depending on need) before getting the access links regardless of whether we prefer to download the data and work on a local machine or whether we want to access the data in the cloud (from a cloud workspace). These can be useful data reduction services that support a faster time to science.\nHarmony\nHarmony allows you to seamlessly analyze Earth observation data from different NASA data centers. These services (API endpoints) provide data reduction (e.g. subsetting) and transfromation services (e.g. convert netCDF data to Zarr cloud optimized format).\n\n\n\nFigure caption: Leverage Harmony cloud-based data transformation services\n\n\nWhen you click the final green Download button, the links provided are to data that had been transformed based on our selections on the previous screen (here chosing to use the Harmony service to reformat the data to Zarr). These data are staged for us in an S3 bucket in AWS, and we can use the s3:// links to access those specific data. This service also provides STAC access links. This particular example is applicable if your workflow is in the AWS us-west-2 region.\n\n\n\nFigure caption: Harmony-staged data in S3\n\n\n\n\n\nStep 5. Integrate file links into programmatic workflow, locally or in the AWS cloud.\nIn tutorial 3 Direct Data Access, we will work programmatically in the cloud to access datasets of interest, to get us set up for further scientific analysis of choice. There are several ways to do this. One way to connect the search part of the workflow we just did in Earthdata Search to our next steps working in the cloud is to simply copy/paste the s3:// links provides in Step 4 above into a JupyterHub notebook or script in our cloud workspace, and continue the data analysis from there.\nOne could also copy/paste the s3:// links and save them in a text file, then open and read the text file in the notebook or script in the JupyterHub in the cloud.\nTutorial 3 will pick up from here and cover these next steps in more detail." + "text": "The following tutorials are meant to present example workflows with NASA Earthdata in the cloud from a variety of use cases. The tutorials exemplify the advantages of working with NASA Earthdata in the cloud and aim to provide a starting point for exploring what your workflows could look like in the cloud.\n\n\nEarthdata Cloud Clinic - Tutorial\nTwo different direct cloud access & subsetting options available in the Earthdata Cloud\n\n\nSea Surface Temperature and Wind Speeds during a Hurricane - Tutorial\nObtaining NASA Earthdata from different disciplines and data centers\n\n\nSea Level Rise - Tutorial\nUsing the earthaccess Python package for quick sea level rise analysis in the cloud\n\n\nDistribution of Carbon Stocks in a Protected Area - Tutorial\nUsing earthaccess Python package and OPeNDAP to access predictions of the aboveground biomass density\n\n\nObserving Seasonality in Agricultural Areas - Tutorial\nWorking with HLS Landsat and Sentinel-2 data products in the cloud\n\n\nICESat-2 and Landsat cloud access and data integration - Tutorial\nAccessing multiple large datasets in the cloud\n\n\nPairing Cloud and non-Cloud Data - Tutorial\nAccessing and harmonizing data located within and outside of the NASA Earthdata Cloud\n\n\nearthaccess Python Library Demo - Tutorial\n\n\nUsing OPeNDAP in the Cloud - Tutorial\n\n\nPygeoweaver Workflow Demo - Tutorial\n\nWe also have links to past workshops. Tutorials in these past workshops are developed to teach open science and Cloud workflows for specific audiences. They are a snapshot in time as workflows with NASA Earthdata Cloud emerge and evolve." }, { - "objectID": "how-tos/aggregate.html", - "href": "how-tos/aggregate.html", - "title": "Aggregate data", + "objectID": "tutorials/Earthdata-cloud-clinic.html", + "href": "tutorials/Earthdata-cloud-clinic.html", + "title": "NASA Earthdata Cloud Clinic", "section": "", - "text": "Coming soon!" - }, - { - "objectID": "how-tos/aggregate.html#how-do-i-calculate-a-climatology", - "href": "how-tos/aggregate.html#how-do-i-calculate-a-climatology", - "title": "Aggregate data", - "section": "How do I calculate a climatology?", - "text": "How do I calculate a climatology?" + "text": "Welcome to the NASA Earthdata Cloud Clinic!\nWe will focus on NASA Earthdata search & access in the cloud.\n\nWe will use earthaccess for data search and direct cloud access, followed by xarray for subsetting. Both are open source python libraries. We will also discover data using Earthdata Search.\nWe will be accessing data directly from the NASA Earthdata Cloud hosted in Amazon Web Services (AWS) Cloud, specifically in the us-west-2 region. This shared compute environment (JupyterHub) is also running in the same location in the AWS Cloud. We will then load the data into Python as an xarray dataset.\nSee second part of this notebook for examples using subsetting servises to downselect and work with data for your use case.\nSee the bottom of the notebook for additional resources, including several tutorials that served as a foundation for this clinic.\n\n\n\nIn this example we will use the earthaccess library to search for data collections from NASA Earthdata. earthaccess is a Python library that simplifies data discovery and access to NASA Earth science data by providing an abstraction layer for NASA’s Common Metadata Repository (CMR) API Search API. The library makes searching for data more approachable by using a simpler notation instead of low level HTTP queries. earthaccess takes the trouble out of Earthdata Login authentication, makes search easier, and provides a stream-line way to download or stream search results into an xarray object for easy data access. It can be used on and off the cloud.\nFor more on earthaccess visit the earthaccess documentation site. Be aware that earthaccess is under active development, and your use and feedback help improve it!\n\n\n\nIn addition to directly accessing the files archived and distributed by each of the NASA DAACs, many datasets also support services that allow us to customize the data via subsetting, reformatting, reprojection/regridding, and file aggregation. What does subsetting mean? Here’s a generalized graphic of what we mean.\n\n\n\n\n“direct cloud access” goes by several other names including “direct S3 access”, “direct access”, “direct in-region access”, “in-cloud data streaming”. And “subsetting” is also called “transformation”." }, { - "objectID": "how-tos/reproject_regrid.html", - "href": "how-tos/reproject_regrid.html", - "title": "How do I reproject and resample a data file?", + "objectID": "tutorials/Earthdata-cloud-clinic.html#summary", + "href": "tutorials/Earthdata-cloud-clinic.html#summary", + "title": "NASA Earthdata Cloud Clinic", "section": "", - "text": "Coming Soon: brief introduction. Define reprojecting & resampling" + "text": "Welcome to the NASA Earthdata Cloud Clinic!\nWe will focus on NASA Earthdata search & access in the cloud.\n\nWe will use earthaccess for data search and direct cloud access, followed by xarray for subsetting. Both are open source python libraries. We will also discover data using Earthdata Search.\nWe will be accessing data directly from the NASA Earthdata Cloud hosted in Amazon Web Services (AWS) Cloud, specifically in the us-west-2 region. This shared compute environment (JupyterHub) is also running in the same location in the AWS Cloud. We will then load the data into Python as an xarray dataset.\nSee second part of this notebook for examples using subsetting servises to downselect and work with data for your use case.\nSee the bottom of the notebook for additional resources, including several tutorials that served as a foundation for this clinic.\n\n\n\nIn this example we will use the earthaccess library to search for data collections from NASA Earthdata. earthaccess is a Python library that simplifies data discovery and access to NASA Earth science data by providing an abstraction layer for NASA’s Common Metadata Repository (CMR) API Search API. The library makes searching for data more approachable by using a simpler notation instead of low level HTTP queries. earthaccess takes the trouble out of Earthdata Login authentication, makes search easier, and provides a stream-line way to download or stream search results into an xarray object for easy data access. It can be used on and off the cloud.\nFor more on earthaccess visit the earthaccess documentation site. Be aware that earthaccess is under active development, and your use and feedback help improve it!\n\n\n\nIn addition to directly accessing the files archived and distributed by each of the NASA DAACs, many datasets also support services that allow us to customize the data via subsetting, reformatting, reprojection/regridding, and file aggregation. What does subsetting mean? Here’s a generalized graphic of what we mean.\n\n\n\n\n“direct cloud access” goes by several other names including “direct S3 access”, “direct access”, “direct in-region access”, “in-cloud data streaming”. And “subsetting” is also called “transformation”." }, { - "objectID": "how-tos/reproject_regrid.html#reproject-with-harmony", - "href": "how-tos/reproject_regrid.html#reproject-with-harmony", - "title": "How do I reproject and resample a data file?", - "section": "Reproject with Harmony", - "text": "Reproject with Harmony" + "objectID": "tutorials/Earthdata-cloud-clinic.html#learning-objectives", + "href": "tutorials/Earthdata-cloud-clinic.html#learning-objectives", + "title": "NASA Earthdata Cloud Clinic", + "section": "Learning Objectives", + "text": "Learning Objectives\n\nUtilize the earthaccess python library to search for data using spatial and temporal filters and explore search results\nStream data (i.e. perform in-region direct access of data) from an Amazon Simple Storage Service (S3) bucket where NASA Earthdata data is archived into our own cloud workspace, here in the Jupyter Lab/Notebook.\nExtract variables and spatial slices from an xarray dataset\nPlot data\nConceptualize data subsetting services provided by NASA Earthdata, including Harmony\nPlot a polygon geojson file with a basemap using geoviews" }, { - "objectID": "how-tos/reproject_regrid.html#reproject-netcdf4", - "href": "how-tos/reproject_regrid.html#reproject-netcdf4", - "title": "How do I reproject and resample a data file?", - "section": "Reproject netcdf4", - "text": "Reproject netcdf4" + "objectID": "tutorials/Earthdata-cloud-clinic.html#prerequisites", + "href": "tutorials/Earthdata-cloud-clinic.html#prerequisites", + "title": "NASA Earthdata Cloud Clinic", + "section": "Prerequisites", + "text": "Prerequisites\nFirst we’ll import python packages and set our authentication that will be used for both of our access and subsetting methods.\nThis tutorial is meant to be run in the AWS cloud in the us-west-2 region. You’ll need to be aware that data in NASA’s Earthdata Cloud reside in Amazon Web Services (AWS) Simple Storage Service (S3) buckets. Access is provided via temporary credentials; this free access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region. While this compute location is required for direct S3 access, all data in Earthdata Cloud are still freely available via download.\n\nImport Required Packages\n\n# Suppress warnings\nimport warnings\nwarnings.simplefilter('ignore')\nwarnings.filterwarnings('ignore')\n\n# Search & Access\nimport earthaccess \nfrom pprint import pprint\nimport xarray as xr\nimport hvplot.xarray #plot\n\n# Harmony services\nimport geopandas as gpd\nimport geoviews as gv\ngv.extension('bokeh', 'matplotlib')\nfrom harmony import BBox, Client, Collection, Request, LinkType\nimport datetime as dt\nimport s3fs\n%matplotlib inline\n\n\n\nAuthentication for NASA Earthdata\nAn Earthdata Login account is required to access data from the NASA Earthdata system. If you don’t already have one, visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\nThe first step is to get the correct authentication that will allow us to get cloud-hosted data from NASA. This is all done through Earthdata Login. We can use the earthaccess library here, where the login method also gets the correct AWS credentials.\nThe first time you run this it will ask you for your Earthdata Login username and password, and stores it in a .netrc file. After that, each time you authenticate with auth = earthaccess.login() it will log you in automatically.\n\n# auth = earthaccess.login(strategy=\"interactive\", persist=True)\nauth = earthaccess.login()" }, { - "objectID": "how-tos/reproject_regrid.html#reproject-geotiff", - "href": "how-tos/reproject_regrid.html#reproject-geotiff", - "title": "How do I reproject and resample a data file?", - "section": "Reproject GeoTIFF", - "text": "Reproject GeoTIFF" + "objectID": "tutorials/Earthdata-cloud-clinic.html#earthaccess-xarray", + "href": "tutorials/Earthdata-cloud-clinic.html#earthaccess-xarray", + "title": "NASA Earthdata Cloud Clinic", + "section": "1. earthaccess + xarray", + "text": "1. earthaccess + xarray\nearthaccess python library is an open-source library to simplify Earthdata Cloud search and access.\n\nSearch for data\nThere are multiple keywords we can use to discovery data from collections such as short_name, concept_id, and doi. The table below contains the short_name for some collections we are interested in for other exercises. Each of these can be used to search for data or information related to the collection we are interested in.\n\n\n\n\n\n\n\n\nShortname\nDescription\nExample Temporal/Spatial parameters\n\n\n\n\nSEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205\ngridded Sea Surface Height Anomalies (SSHA) above a mean sea surface, on a 1/6th degree grid every 5 days.\ntemporal=(“2021-07-01”, “2021-09-30”)\n\n\nMUR25-JPL-L4-GLOB-v04.2\nMUR Sea Surface Temperature\ntemporal=(“2023-07-01”, “2023-07-03”)\n\n\nSMAP_RSS_L3_SSS_SMI_8DAY-RUNNINGMEAN_V5\nSMAP Sea Surface Salinity 8 day running mean\ntemporal=(“2019-08-01T00:00:00”, “2019-10-01T00:00:00”)\n\n\n\nBut wait…You may be asking “how can we find the short_name for collections not in the table above?”.\n–> Let’s take a quick detour and head to Earthdata Search GUI to gather more information about our dataset of interest. The dataset “short name” can be found by clicking on the Info button on our collection search result, and we can paste that into a python variable.\n(Side Note: Both earthaccess python libray and the Earthdata Search (GUI) leverage the Common Metadata Repository (CMR) API to search for collections and granules.)\nHere we use the search_data function of earthaccess to query based on the short_name of interest, as well as other paramters such as temporal range:\n\ndata_name = \"SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205\"\n\nresults = earthaccess.search_data(\n short_name=data_name,\n cloud_hosted=True,\n temporal=(\"2021-07-01\", \"2021-09-30\"),\n)\n\nAccording to PO.DAAC’s dataset landing page, gridded Sea Surface Height Anomalies (SSHA) above a mean sea surface are provided. The data are produced on a 1/6th degree grid every 5 days.\nWe can discover more information about the matching files:\n\npprint(results[0])\n\n\n\nAccess data\nSince we are working in the AWS us-west-2 region, we can stream data directly to xarray. We are using the open_mfdataset() (multi-file) method, which is required when using earthaccess.\n(Tips: To open a single file, if troubleshooting for example: ds = xr.open_dataset(earthaccess.open(results)[0]))\n\nds = xr.open_mfdataset(earthaccess.open(results))\nds\n\n\nPlot the data\nLet’s make a quick interactive plot of the data using an open source tool call hvplot. Because our data is 3d and has a time component, we can also preview the data over time, using the slider of hvplot.\n\nds.SLA.hvplot.image(x='Longitude', y='Latitude', cmap='RdBu', clim=(-2,2), title=\"Sea Level Anomaly Estimate (m)\")\n\n\n\n\nSubset data\nWe can create a subsetted xarray dataset by extracting the SLA variable and slicing the dataset by a smaller area of interest near the state of Baja, Mexico.\n\nds_subset = ds['SLA'].sel(Latitude=slice(15.8, 35.9), Longitude=slice(234.5,260.5)) \nds_subset\n\n\nds_subset.hvplot.image(x='Longitude', y='Latitude', cmap='RdBu', clim=(-0.5, 0.5), title=\"Sea Level Anomaly Estimate (m)\")" }, { - "objectID": "how-tos/reproject_regrid.html#resample-with-hamony", - "href": "how-tos/reproject_regrid.html#resample-with-hamony", - "title": "How do I reproject and resample a data file?", - "section": "Resample with Hamony", - "text": "Resample with Hamony" + "objectID": "tutorials/Earthdata-cloud-clinic.html#exercise", + "href": "tutorials/Earthdata-cloud-clinic.html#exercise", + "title": "NASA Earthdata Cloud Clinic", + "section": "Exercise", + "text": "Exercise\nLet’s explore other datasets from the table above\n\nMUR SST\n\ndata_name = \"MUR25-JPL-L4-GLOB-v04.2\"\n\nsst_results = earthaccess.search_data(\n short_name=data_name,\n cloud_hosted=True,\n temporal=(\"2023-07-01\", \"2023-07-03\"), \n)\n\n\npprint(sst_results[0])\n\n\nds = xr.open_mfdataset(earthaccess.open(sst_results))\nds\n\n\nds.sst_anomaly.hvplot.image(x='lon', y='lat', cmap='RdBu', clim=(-5,5), title=\"Sea Surface Temperature Anomaly (K)\")\n\n\n\nSMAP Sea Surface Salinity\n\ndata_name=\"SMAP_RSS_L3_SSS_SMI_8DAY-RUNNINGMEAN_V5\"\n\nsmap_results = earthaccess.search_data(\n short_name=data_name,\n cloud_hosted=True,\n temporal=(\"2019-08-01T00:00:00\", \"2019-10-01T00:00:00\"),\n)\n\n\npprint(smap_results[0])\n\n\nds = xr.open_mfdataset(earthaccess.open(smap_results))\nds\n\n\nds.sss_smap.hvplot.image(x='lon', y='lat', cmap='plasma', clim=(20, 32), title=\"SMAP_RSS_L3_SSS_SMI_8DAY-RUNNINGMEAN_V5\")\n\n\nds_subset = ds['sss_smap'].sel(lat=slice(61, 75), lon=slice(160,170)) \nds_subset\n\n\nds_subset.hvplot.image(x='lon', y='lat', cmap='plasma', clim=(0, 45), title=\"SMAP_RSS_L3_SSS_SMI_8DAY-RUNNINGMEAN_V5\")\n\n\n\n\nSpatial search tip:\nTo narrow down your search based on a geographic bounding box, we can pass the bounding_box= paramters to earthaccess, as in the example below: #bounding_box=(lower_left_lon, lower_left_lat, upper_right_lon, upper_right_lat) bounding_box=(-99, 18.19232, -78.85547, 31.23754)" }, { - "objectID": "how-tos/reproject_regrid.html#resample-netcdf4", - "href": "how-tos/reproject_regrid.html#resample-netcdf4", - "title": "How do I reproject and resample a data file?", - "section": "Resample netcdf4", - "text": "Resample netcdf4" + "objectID": "tutorials/Earthdata-cloud-clinic.html#harmony-py-xarray", + "href": "tutorials/Earthdata-cloud-clinic.html#harmony-py-xarray", + "title": "NASA Earthdata Cloud Clinic", + "section": "2. Harmony-py + xarray", + "text": "2. Harmony-py + xarray\nBonus cloud clinic topic that we won’t cover live together: subsetting with Earthdata Harmony API services\nWe can also use the Harmony-py python library for direct cloud access & data subsetting: * Requesting a subset of data from the GHRSST Level 4 MUR Global Foundation Sea Surface Temperature Analysis (v4.1) dataset using a vector-based geospatial file. * This dataset can also be viewed in Earthdata Search.\nharmony-py + xarray\nWe will be accessing a subsetted NetCDF-4 file using Transformation Services in the Cloud.\nUsing Earthdata Search, we can find datasets that support these services using the “Customizable” filter.\nWe will find, request, and open customized data using Harmony, below.\n\nDefine area of interest\nFirst, use geopandas to read in a geojson file containing a polygon feature over the Gulf of Mexico. The geojson file is found in the ~/data directory.\n\ngeojson_path = './data/gulf.json'\ngdf = gpd.read_file(geojson_path) #Return a GeoDataFrame object\n\nWe can plot the polygon using the geoviews package that we imported as gv with ‘bokeh’ and ‘matplotlib’ extensions. The following has reasonable width, height, color, and line widths to view our polygon when it is overlayed on a base tile map. We can view this with the * operator.\n\nbase = gv.tile_sources.EsriImagery.opts(width=650, height=500)\nocean_map = gv.Polygons(gdf).opts(line_color='yellow', line_width=5, color=None)\nbase * ocean_map\n\n\n\nCreate a subset request\nHere we’ll create a request for a spatial subset of data.\nFirst, we need to create a Harmony Client, which is what we will interact with to submit and inspect a data request to Harmony, as well as to retrieve results.\nWhen creating the Client, we need to provide Earthdata Login credentials. This basic line below assumes that we have a .netrc available. See the Earthdata Cloud Cookbook appendix for more information on Earthdata Login and netrc setup.\n\nharmony_client = Client()\n\nSee the harmony-py documentation for details on how to construct your request.\n\nsst_short_name=\"MUR-JPL-L4-GLOB-v4.1\"\n\nrequest = Request(\n collection=Collection(id=sst_short_name),\n shape=geojson_path,\n temporal={\n 'start': dt.datetime(2021, 8, 1, 1),\n 'stop': dt.datetime(2021, 8, 1, 2) \n },\n)\n\n\n\nSubmit a subset request\nNow that the request is created, we can now submit it to Harmony using the Harmony Client object. A job id is returned, which is a unique identifier that represents the submitted request.\n\njob_id = harmony_client.submit(request)\njob_id\n\n\n\nCheck request status\nDepending on the size of the request, it may be helpful to wait until the request has completed processing before the remainder of the code is executed. The wait_for_processing() method will block subsequent lines of code while optionally showing a progress bar.\n\nharmony_client.wait_for_processing(job_id, show_progress=True)\n\nWe can view Harmony job response and output URLs Once the data request has finished processing, we can view details on the job that was submitted to Harmony, including the API call to Harmony, and informational messages on the request if available.\nresult_json() calls wait_for_processing() and returns the complete job in JSON format once processing is complete.\n\ndata = harmony_client.result_json(job_id)\npprint(data)\n\n\n\nAccess data\nJust like above, the subsetted outputs produced by Harmony can be accessed directly from the cloud.\nFirst, we’ll retrieve list of output URLs.\nThe result_urls() method calls wait_for_processing() and returns a list of the processed data URLs once processing is complete. You may optionally show the progress bar as shown below.\n\nresults = harmony_client.result_urls(job_id, link_type=LinkType.s3)\nurls = list(results)\nurl = urls[0]\nprint(url)\n\nUsing aws_credentials you can retrieve the credentials needed to access the Harmony s3 staging bucket and its contents.\n\ncreds = harmony_client.aws_credentials()\n\n\n\nOpen staged files with s3fs and xarray\nWe use the AWS s3fs package to create a file system that can then be read by xarray:\n\ns3_fs = s3fs.S3FileSystem(\n key=creds['aws_access_key_id'],\n secret=creds['aws_secret_access_key'],\n token=creds['aws_session_token'],\n client_kwargs={'region_name':'us-west-2'},\n)\n\nNow that we have our s3 file system set, including our declared credentials, we’ll use that to open the url, and read in the file through xarray. This extra step is needed because xarray cannot open the S3 location directly. Instead, the S3 file object is passed to xarray, in order to then open the dataset.\n\nf = s3_fs.open(url, mode='rb')\nds = xr.open_dataset(f)\nds\n\nAs before, we use the xarray built in plotting function to create a simple plot along the x and y dimensions of the dataset. We can see that the data are subsetted to our polygon:\n\nds.analysed_sst.plot();" }, { - "objectID": "how-tos/reproject_regrid.html#resample-geotiff", - "href": "how-tos/reproject_regrid.html#resample-geotiff", - "title": "How do I reproject and resample a data file?", - "section": "Resample GeoTIFF", - "text": "Resample GeoTIFF" + "objectID": "tutorials/Earthdata-cloud-clinic.html#additional-resources", + "href": "tutorials/Earthdata-cloud-clinic.html#additional-resources", + "title": "NASA Earthdata Cloud Clinic", + "section": "Additional Resources", + "text": "Additional Resources\n\nTutorials\nThis clinic was based off of several notebook tutorials including those presented during past workshop events, along with other materials co-created by the NASA Openscapes mentors: * 2021 Earthdata Cloud Hackathon * 2021 AGU Workshop * Accessing and working with ICESat-2 data in the cloud * Analyzing Sea Level Rise Using Earth Data in the Cloud\n\n\nCloud services\nThe examples used in the clinic provide an abbreviated and simplified workflow to explore access and subsetting options available through the Earthdata Cloud. There are several other options that can be used to interact with data in the Earthdata Cloud including:\n\nOPeNDAP\n\nHyrax provides direct access to subsetting of NASA data using Python or your favorite analysis tool\nTutorial highlighting OPeNDAP usage: https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/working-locally/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html\n\nZarr-EOSDIS-Store\n\nThe zarr-eosdis-store library allows NASA EOSDIS Collections to be accessed efficiently by the Zarr Python library, provided they have a sidecar DMR++ metadata file generated.\nTutorial highlighting this library’s usage: https://nasa-openscapes.github.io/2021-Cloud-Hackathon/tutorials/09_Zarr_Access.html\n\n\n\n\nSupport\n\nEarthdata Forum\n\nUser Services and community support for all things NASA Earthdata, including Earthdata Cloud\n\nEarthdata Webinar series\n\nWebinars from DAACs and other groups across EOSDIS including guidance on working with Earthdata Cloud\nSee the Earthdata YouTube channel for more videos" }, { - "objectID": "when-to-cloud.html", - "href": "when-to-cloud.html", - "title": "When To ‘Cloud’", + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html", + "title": "Observing Seasonality in Agricultural Areas", "section": "", - "text": "Cloud adoption often has a steep learning curve and can feel overwhelming. There are times when using the cloud is effective and times when the download model is more appropriate. Here we aim to help you decide what’s best for your use case.\n\nWhat does it mean to be in The Cloud?\n\n\n\nImage by Alexis Hunzinger, GES DISC\n\n\nAt a basic level, “The Cloud” is somewhere that isn’t your computer. We all interact with data and services and that live in “The Cloud” in our daily lives. When we store photos in iCloud or Google accounts instead of on our cell phones, we are using cloud storage. When we watch movies and tv shows on streaming services like Netflix or Hulu, we are using the cloud. In these cases, we are interacting with “the cloud” without knowing it, though we, the user, are not in “the cloud”.\n\n\n\nImage by Alexis Hunzinger, GES DISC\n\n\nIf you use services like computing and storage, provided by a cloud service provider (Amazon Web Services, Microsoft Azure, Google Cloud Platform, etc.), then you are in “the cloud”. Remember, “the cloud” is somewhere that isn’t your computer. The storage location is elsewhere and the machines and processing is elsewhere.\n\n\n\nImage by Alexis Hunzinger, GES DISC\n\n\nThe following guidance is specific to NASA Earthdata and the particular cloud it is stored in, Amazon Web Services (AWS). While some services provided by NASA data archives are cloud-based and you interact with, the guidance below refers to using compute and storage services provided by AWS that allow a user to work closely, or in the same cloud, as the data.\n\n\n\nImage by Alexis Hunzinger, GES DISC\n\n\n\n\nQuestions to ask yourself\nUse the following questions to guide your decisionmaking process about using cloud access and computing:\n\nWhat is the data volume?\nHow long will it take to download?\nCan you store all that data (cost and space)?\nDo you have the computing power for processing?\nDoes your team need a common computing environment?\nDo you need to share data at each step or just an end product?\nIs the action I want to take an existing service? (i.e. subsetting a dataset)\n\nSource: Data strategies for Future Us by Andy Barrett\n\n\nTo Cloud…\nTODO: list example cases here: identify and describe what makes cloud usage more benficial for the team/workflow\nFind more discussion on cloud-based solutions for geoscience in this Medium article by Ryan Abernathy and Joe Hamman.\n\n\nNot To Cloud…\nTODO: list examples cases here: identify and describe what makes the non-option more beneficial for the team/workflow\n\n\nChallenges\nIt is important to be aware of the drawbacks and challenges associated with working in the cloud. Feedback from early cloud adopters are summarized here:\n\n“I don’t have the time or energy to learn so many new concepts.”\n“Major challenge: budgeting for cloud in a proposal.”\n“My workflow isn’t parallelizable, the file format isn’t cloud-optimized, I’d rather download everything I need and perform analysis locally.”\n“[On the preference to perform analysis locally] …this is fine if data wasn’t getting bigger!”\n“[The cloud] is faster, but only by a factor of 2. Our setup and data granules aren’t structured to take advantage of a faster cloud.”\n“Worried about doing things ‘on my own’, outside of a cloud-hosted JupyterHub.”\n“How does using data from AWS work with GEE?”\n\nSource: The Cloud: Obstacles and Barriers Encountered by Users (AGU 2023 - Alexis Hunzinger, Christopher Battisto, Allison Alcott, Binita KC)\n\n\nConsiderations\nWe are now accustomed to living in a highly digital world, separated from physical reminders of the technology we use. No longer do we access documents from a row of filing cabinets, we now store them in cloud-based archives (e.g. Google Docs). We run code on high-performance computing clusters, removed from the whirring and warmth generated by servers that live away from our desks. The following are considerations for using powerful cloud-based resources:\n\nRemoved from physical signs of energy usage (heat, noise) of supercomputers/servers\nPhysical location of “the cloud”, whose land is it on? What resources is it using?\nEnvironmental impacts of large server farms\nConsider testing locally before migrating a workflow to the cloud\n\nDoes cloud storage and computing increase equity?\n\nYes! Cloud is a solution for equity, as it provides under-resourced organizations (e.g. small universities, international groups) access to equipment they don’t own or maintain.\nNo! Larger organizations, who already own and maintain equipment for storage and computing, often have the budget to also use cloud storage and computing. This increases their pool of resources, giving them a larger advantage." + "text": "This tutorial was developed to examine changes in enhanced vegetation index (EVI) over an agricultural region in northern California. The goal of the project is to observe HLS-derived mean EVI over these regions without downloading the entirety of the HLS source data. In this notebook we will extract an EVI timeseries from Harmonized Landsat Sentinel-2 (HLS) data in the Cloud using the earthaccess and rioxarray libraries. This tutorial will show how to find the HLS data available in the cloud for our specific time period, bands (layers), and region of interest. After finding the desired data, we will load subsets of the cloud optimized geotiffs (COGs) into a Jupyter Notebook directly from the cloud, and calculate EVI. After calculating EVI we will save and stack the time series, visualize it, and export a CSV of EVI statistics for the region.\nThe Harmonized Landsat Sentinel-2 (HLS) project produces seamless, harmonized surface reflectance data from the Operational Land Imager (OLI) and Multi-Spectral Instrument (MSI) aboard Landsat and Sentinel-2 Earth-observing satellites, respectively. The aim is to produce seamless products with normalized parameters, which include atmospheric correction, cloud and cloud-shadow masking, geographic co-registration and common gridding, normalized bidirectional reflectance distribution function, and spectral band adjustment. This will provide global observation of the Earth’s surface every 2-3 days with 30 meter spatial resolution. One of the major applications that will benefit from HLS is agriculture assessment and monitoring, which is used as the use case for this tutorial.\nNASA’s Land Processes Distributed Active Archive Center (LP DAAC) archives and distributes HLS products in the LP DAAC Cumulus cloud archive as Cloud Optimized GeoTIFFs (COG). This tutorial will demonstrate Because these data are stored as COGs, this tutorial will teach users how to load subsets of individual files into memory for just the bands you are interested in–a paradigm shift from the more common workflow where you would need to download a .zip/HDF file containing every band over the entire scene/tile. This tutorial covers how to process HLS data (calculate EVI), visualize, and “stack” the scenes over a region of interest into an xarray data array, calculate statistics for an EVI time series, and export as a comma-separated values (CSV) file–providing you with all of the information you need for your area of interest without having to download the source data file. The Enhanced Vegetation Index (EVI), is a vegetation index similar to NDVI that has been found to be more sensitive to ground cover below the vegetated canopy and saturates less over areas of dense green vegetation." }, { - "objectID": "leading-workshops/index.html", - "href": "leading-workshops/index.html", - "title": "Leading Workshops", - "section": "", - "text": "This section, under development, will include information for workshop prep, setup, and hosting." - }, - { - "objectID": "leading-workshops/index.html#using-the-openscapes-2i2c-hub-in-a-workshop", - "href": "leading-workshops/index.html#using-the-openscapes-2i2c-hub-in-a-workshop", - "title": "Leading Workshops", - "section": "Using the Openscapes 2i2c Hub in a workshop", - "text": "Using the Openscapes 2i2c Hub in a workshop\n\nCheck with Luis that the Hub image has the packages you need\nReach out to 2i2c a month in advance via email support at 2i2c.freshdesk.com (example below) to tell them about the workshop date, start and end times, # of participants, anticipated level of resources to be used.\nAdd participants to the 2i2c Hub via a GitHub Team\n\n\n\n\n\n\n\nExample email to 2i2c for a workshop\n\n\n\n\n\nHello,\nI wanted to reach out to the support desk to let you know that Openscapes will have a workshop using Python 2i2c instances for about 50 attendees.\nTitle: Data Access Workshop for NASA’s SWOT Satellite\nDate: February 13, 2024\nDuration/Time: half day, 9:00 am-12:30 pm HST (Honolulu, HI).\nExpected Usage: 3.7 GB per person (~50 people)\nThank you!\nCassie\n\n\n\n\nHow to Add Folks to the 2i2c Hub" - }, - { - "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html", - "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html", - "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", - "section": "", - "text": "In this notebook, we will access monthly sea surface height from ECCO V4r4 (10.5067/ECG5D-SSH44). The data are provided as a time series of monthly netCDFs on a 0.5-degree latitude/longitude grid.\nWe will access the data from inside the AWS cloud (us-west-2 region, specifically) and load a time series made of multiple netCDF datasets into an xarray dataset. This approach leverages S3 native protocols for efficient access to the data." - }, - { - "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#summary", - "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#summary", - "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#overview", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#overview", + "title": "Observing Seasonality in Agricultural Areas", "section": "", - "text": "In this notebook, we will access monthly sea surface height from ECCO V4r4 (10.5067/ECG5D-SSH44). The data are provided as a time series of monthly netCDFs on a 0.5-degree latitude/longitude grid.\nWe will access the data from inside the AWS cloud (us-west-2 region, specifically) and load a time series made of multiple netCDF datasets into an xarray dataset. This approach leverages S3 native protocols for efficient access to the data." + "text": "This tutorial was developed to examine changes in enhanced vegetation index (EVI) over an agricultural region in northern California. The goal of the project is to observe HLS-derived mean EVI over these regions without downloading the entirety of the HLS source data. In this notebook we will extract an EVI timeseries from Harmonized Landsat Sentinel-2 (HLS) data in the Cloud using the earthaccess and rioxarray libraries. This tutorial will show how to find the HLS data available in the cloud for our specific time period, bands (layers), and region of interest. After finding the desired data, we will load subsets of the cloud optimized geotiffs (COGs) into a Jupyter Notebook directly from the cloud, and calculate EVI. After calculating EVI we will save and stack the time series, visualize it, and export a CSV of EVI statistics for the region.\nThe Harmonized Landsat Sentinel-2 (HLS) project produces seamless, harmonized surface reflectance data from the Operational Land Imager (OLI) and Multi-Spectral Instrument (MSI) aboard Landsat and Sentinel-2 Earth-observing satellites, respectively. The aim is to produce seamless products with normalized parameters, which include atmospheric correction, cloud and cloud-shadow masking, geographic co-registration and common gridding, normalized bidirectional reflectance distribution function, and spectral band adjustment. This will provide global observation of the Earth’s surface every 2-3 days with 30 meter spatial resolution. One of the major applications that will benefit from HLS is agriculture assessment and monitoring, which is used as the use case for this tutorial.\nNASA’s Land Processes Distributed Active Archive Center (LP DAAC) archives and distributes HLS products in the LP DAAC Cumulus cloud archive as Cloud Optimized GeoTIFFs (COG). This tutorial will demonstrate Because these data are stored as COGs, this tutorial will teach users how to load subsets of individual files into memory for just the bands you are interested in–a paradigm shift from the more common workflow where you would need to download a .zip/HDF file containing every band over the entire scene/tile. This tutorial covers how to process HLS data (calculate EVI), visualize, and “stack” the scenes over a region of interest into an xarray data array, calculate statistics for an EVI time series, and export as a comma-separated values (CSV) file–providing you with all of the information you need for your area of interest without having to download the source data file. The Enhanced Vegetation Index (EVI), is a vegetation index similar to NDVI that has been found to be more sensitive to ground cover below the vegetated canopy and saturates less over areas of dense green vegetation." }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#requirements", - "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#requirements", - "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#requirements", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#requirements", + "title": "Observing Seasonality in Agricultural Areas", "section": "Requirements", - "text": "Requirements\n\n1. AWS instance running in us-west-2\nNASA Earthdata Cloud data in S3 can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region.\n\n\n2. Earthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\n3. netrc File\nYou will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. For additional information see: Authentication for NASA Earthdata." + "text": "Requirements\n\nA NASA Earthdata Login account is required to download the data used in this tutorial. You can create an account at the link provided.\nYou will will also need to have a netrc file set up in your home directory in order to successfully run the code below. A code chunk in a later section provides a way to do this, or you can check out the setup_intstructions.md." }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#learning-objectives", - "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#learning-objectives", - "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#learning-objectives", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#learning-objectives", + "title": "Observing Seasonality in Agricultural Areas", "section": "Learning Objectives", - "text": "Learning Objectives\n\nhow to retrieve temporary S3 credentials for in-region direct S3 bucket access\nhow to define a dataset of interest and find netCDF files in S3 bucket\nhow to perform in-region direct access of ECCO_L4_SSH_05DEG_MONTHLY_V4R4 data in S3\nhow to plot the data" - }, - { - "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#import-packages", - "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#import-packages", - "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", - "section": "Import Packages", - "text": "Import Packages\n\nimport os\nimport requests\nimport s3fs\nimport xarray as xr\nimport hvplot.xarray" - }, - { - "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#get-temporary-aws-credentials", - "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#get-temporary-aws-credentials", - "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", - "section": "Get Temporary AWS Credentials", - "text": "Get Temporary AWS Credentials\nDirect S3 access is achieved by passing NASA supplied temporary credentials to AWS so we can interact with S3 objects from applicable Earthdata Cloud buckets. For now, each NASA DAAC has different AWS credentials endpoints. Below are some of the credential endpoints to various DAACs:\n\ns3_cred_endpoint = {\n 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n 'gesdisc': 'https://data.gesdisc.earthdata.nasa.gov/s3credentials',\n 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n 'ornldaac': 'https://data.ornldaac.earthdata.nasa.gov/s3credentials',\n 'ghrcdaac': 'https://data.ghrc.earthdata.nasa.gov/s3credentials'\n}\n\nCreate a function to make a request to an endpoint for temporary credentials. Remember, each DAAC has their own endpoint and credentials are not usable for cloud data from other DAACs.\n\ndef get_temp_creds(provider):\n return requests.get(s3_cred_endpoint[provider]).json()\n\n\ntemp_creds_req = get_temp_creds('podaac')\n#temp_creds_req" - }, - { - "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#set-up-an-s3fs-session-for-direct-access", - "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#set-up-an-s3fs-session-for-direct-access", - "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", - "section": "Set up an s3fs session for Direct Access", - "text": "Set up an s3fs session for Direct Access\ns3fs sessions are used for authenticated access to s3 bucket and allows for typical file-system style operations. Below we create session by passing in the temporary credentials we recieved from our temporary credentials endpoint.\n\nfs_s3 = s3fs.S3FileSystem(anon=False, \n key=temp_creds_req['accessKeyId'], \n secret=temp_creds_req['secretAccessKey'], \n token=temp_creds_req['sessionToken'],\n client_kwargs={'region_name':'us-west-2'})\n\nIn this example we’re interested in the ECCO data collection from NASA’s PO.DAAC in Earthdata Cloud. In this case it’s the following string that unique identifies the collection of monthly, 0.5-degree sea surface height data (ECCO_L4_SSH_05DEG_MONTHLY_V4R4).\n\nshort_name = 'ECCO_L4_SSH_05DEG_MONTHLY_V4R4'\n\n\nbucket = os.path.join('podaac-ops-cumulus-protected/', short_name, '*2015*.nc')\nbucket\n\nGet a list of netCDF files located at the S3 path corresponding to the ECCO V4r4 monthly sea surface height dataset on the 0.5-degree latitude/longitude grid, for year 2015.\n\nssh_files = fs_s3.glob(bucket)\nssh_files" - }, - { - "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#direct-in-region-access", - "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#direct-in-region-access", - "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", - "section": "Direct In-region Access", - "text": "Direct In-region Access\nOpen with the netCDF files using the s3fs package, then load them all at once into a concatenated xarray dataset.\n\nfileset = [fs_s3.open(file) for file in ssh_files]\n\nCreate an xarray dataset using the open_mfdataset() function to “read in” all of the netCDF4 files in one call.\n\nssh_ds = xr.open_mfdataset(fileset,\n combine='by_coords',\n mask_and_scale=True,\n decode_cf=True,\n chunks='auto')\nssh_ds\n\nGet the SSH variable as an xarray dataarray\n\nssh_da = ssh_ds.SSH\nssh_da\n\nPlot the SSH time series using hvplot\n\nssh_da.hvplot.image(y='latitude', x='longitude', cmap='Viridis',).opts(clim=(ssh_da.attrs['valid_min'][0],ssh_da.attrs['valid_max'][0]))" + "text": "Learning Objectives\n\nHow to work with HLS Landsat (HLSL30.002) and Sentinel-2 (HLSS30.002) data products\n\nHow to query and subset HLS data using the earthaccess library\n\nHow to access and work with HLS data" }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#resources", - "href": "appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html#resources", - "title": "Accessing Multiple NetCDF4/HDF5 Files - S3 Direct Access", - "section": "Resources", - "text": "Resources\nDirect access to ECCO data in S3 (from us-west-2)\nData_Access__Direct_S3_Access__PODAAC_ECCO_SSH using CMR-STAC API to retrieve S3 links" + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#data-used", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#data-used", + "title": "Observing Seasonality in Agricultural Areas", + "section": "Data Used", + "text": "Data Used\n\nDaily 30 meter (m) global HLS Sentinel-2 Multi-spectral Instrument Surface Reflectance - HLSS30.002\n\nThe HLSS30 product provides 30 m Nadir normalized Bidirectional Reflectance Distribution Function (BRDF)-Adjusted Reflectance (NBAR) and is derived from Sentinel-2A and Sentinel-2B MSI data products.\n\nScience Dataset (SDS) layers:\n\nB8A (NIR Narrow)\n\nB04 (Red)\n\nB02 (Blue)\n\nFmask (Quality)\n\n\n\nDaily 30 meter (m) global HLS Landsat-8 OLI Surface Reflectance - HLSL30.002\n\nThe HLSL30 product provides 30 m Nadir normalized Bidirectional Reflectance Distribution Function (BRDF)-Adjusted Reflectance (NBAR) and is derived from Landsat-8 OLI data products.\n\nScience Dataset (SDS) layers:\n\nB05 (NIR)\n\nB04 (Red)\n\nB02 (Blue)\n\nFmask (Quality)" }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html", - "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html", - "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", - "section": "", - "text": "In this notebook, we will access monthly sea surface height from ECCO V4r4 (10.5067/ECG5D-SSH44). The data are provided as a time series of monthly netCDFs on a 0.5-degree latitude/longitude grid.\nWe will access a single netCDF file from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataset. This approach leverages S3 native protocols for efficient access to the data." + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#tutorial-outline", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#tutorial-outline", + "title": "Observing Seasonality in Agricultural Areas", + "section": "Tutorial Outline", + "text": "Tutorial Outline\n\nGetting Started\n1.1 Import Packages\n1.2 EarthData Login\n\nFinding HLS Data\n\nAccessing HLS Cloud Optimized GeoTIFFs (COGs) from Earthdata Cloud 3.1 Subset by Band\n3.2 Load COGS into Memory\n3.3 Subset Spatially\n3.4 Apply Scale Factor\n\nProcessing HLS Data\n4.1 Calculate EVI\n4.2 Export to COG\n\nAutomation\n\nStacking HLS Data\n6.1 Open and Stack COGs\n6.2 Visualize Stacked Time Series\n\nExport Statistics" }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#summary", - "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#summary", - "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", - "section": "", - "text": "In this notebook, we will access monthly sea surface height from ECCO V4r4 (10.5067/ECG5D-SSH44). The data are provided as a time series of monthly netCDFs on a 0.5-degree latitude/longitude grid.\nWe will access a single netCDF file from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataset. This approach leverages S3 native protocols for efficient access to the data." + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#getting-started", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#getting-started", + "title": "Observing Seasonality in Agricultural Areas", + "section": "1. Getting Started", + "text": "1. Getting Started\n\n1.1 Import Packages\nImport the required packages.\n\nimport os\nfrom datetime import datetime\nimport requests as r\nimport numpy as np\nimport pandas as pd\nimport geopandas as gp\nfrom skimage import io\nimport matplotlib.pyplot as plt\nfrom osgeo import gdal\nimport rasterio as rio\nimport xarray as xr\nimport rioxarray as rxr\nimport hvplot.xarray\nimport hvplot.pandas\nimport json\nimport panel as pn\nimport geoviews\nimport earthaccess\nfrom pprint import pprint\n\n\n\n\n\n\n\n\n\n\n\n\n\n1.2 Earthdata Login Authentication\nWe will use the earthaccess package for authentication. earthaccess can either createa a new local .netrc file to store credentials or validate that one exists already in you user profile. If you do not have a .netrc file, you will be prompted for your credentials and one will be created.\n\nauth = earthaccess.login()\n\nYou're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 11/06/2023\nUsing .netrc file for EDL" }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#requirements", - "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#requirements", - "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", - "section": "Requirements", - "text": "Requirements\n\n1. AWS instance running in us-west-2\nNASA Earthdata Cloud data in S3 can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region.\n\n\n2. Earthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\n3. netrc File\nYou will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. For additional information see: Authentication for NASA Earthdata." + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#finding-hls-data-using-earthaccess", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#finding-hls-data-using-earthaccess", + "title": "Observing Seasonality in Agricultural Areas", + "section": "2. Finding HLS Data using earthaccess ", + "text": "2. Finding HLS Data using earthaccess \nTo find HLS data, we will use the earthaccess python library to search NASA’s Common Metadata Repository (CMR) for HLS data. We will use an geojson file containing our region of interest (ROI) to search for files that intersect. To do this, we will simplify it to a bounding box. Grab the bounding coordinates from the geopandas object after opening.\nFirst we will read in our geojson file using geopandas\n\nfield = gp.read_file('https://git.earthdata.nasa.gov/projects/LPDUR/repos/hls-tutorial/raw/Field_Boundary.geojson?at=refs%2Fheads%2Fmain')\n\nWe will use the total_bounds property to get the bounding box of our ROI, and add that to a python tuple, which is the expected data type for the bounding_box parameter earthaccess search_data.\n\nbbox = tuple(list(field.total_bounds))\nbbox\n\n(-122.0622682571411, 39.897234301806, -122.04918980598451, 39.91309383703065)\n\n\nWhen searching we can also search a specific time period of interest. Here we search from the beginning of May 2021 to the end of September 2021.\n\ntemporal = (\"2021-05-01T00:00:00\", \"2021-09-30T23:59:59\")\n\nSince the HLS collection contains to products, i.e. HLSL30 and HLSS30, we will include both short names. Search using our constraints and the count = 100 to limit our search to 100 results.\n\nresults = earthaccess.search_data(\n short_name=['HLSL30','HLSS30'],\n bounding_box=bbox,\n temporal=temporal, # 2021-07-15T00:00:00Z/2021-09-15T23:59:59Z\n count=100\n)\n\nGranules found: 78\n\n\nWe can preview these results in a pandas dataframe we want to check the metadata. Note we only show the first 5.\n\npd.json_normalize(results).head(5)\n\n\n\n\n\n\n\n\nsize\nmeta.concept-type\nmeta.concept-id\nmeta.revision-id\nmeta.native-id\nmeta.provider-id\nmeta.format\nmeta.revision-date\numm.TemporalExtent.RangeDateTime.BeginningDateTime\numm.TemporalExtent.RangeDateTime.EndingDateTime\n...\numm.CollectionReference.EntryTitle\numm.RelatedUrls\numm.DataGranule.DayNightFlag\numm.DataGranule.Identifiers\numm.DataGranule.ProductionDateTime\numm.DataGranule.ArchiveAndDistributionInformation\numm.Platforms\numm.MetadataSpecification.URL\numm.MetadataSpecification.Name\numm.MetadataSpecification.Version\n\n\n\n\n0\n198.882758\ngranule\nG2153255128-LPCLOUD\n1\nHLS.S30.T10TEK.2021122T185911.v2.0\nLPCLOUD\napplication/echo10+xml\n2021-10-29T10:47:01.310Z\n2021-05-02T19:13:19.930Z\n2021-05-02T19:13:19.930Z\n...\nHLS Sentinel-2 Multi-spectral Instrument Surfa...\n[{'URL': 'https://data.lpdaac.earthdatacloud.n...\nDay\n[{'Identifier': 'HLS.S30.T10TEK.2021122T185911...\n2021-10-29T10:44:28.000Z\n[{'Name': 'Not provided', 'SizeInBytes': 20854...\n[{'ShortName': 'Sentinel-2A', 'Instruments': [...\nhttps://cdn.earthdata.nasa.gov/umm/granule/v1.6.5\nUMM-G\n1.6.5\n\n\n1\n186.999763\ngranule\nG2153407526-LPCLOUD\n1\nHLS.S30.T10TEK.2021124T184919.v2.0\nLPCLOUD\napplication/echo10+xml\n2021-10-29T14:09:37.033Z\n2021-05-04T19:03:22.997Z\n2021-05-04T19:03:22.997Z\n...\nHLS Sentinel-2 Multi-spectral Instrument Surfa...\n[{'URL': 'https://data.lpdaac.earthdatacloud.n...\nDay\n[{'Identifier': 'HLS.S30.T10TEK.2021124T184919...\n2021-10-29T13:51:48.000Z\n[{'Name': 'Not provided', 'SizeInBytes': 19608...\n[{'ShortName': 'Sentinel-2B', 'Instruments': [...\nhttps://cdn.earthdata.nasa.gov/umm/granule/v1.6.5\nUMM-G\n1.6.5\n\n\n2\n199.237585\ngranule\nG2153881444-LPCLOUD\n1\nHLS.S30.T10TEK.2021127T185919.v2.0\nLPCLOUD\napplication/echo10+xml\n2021-10-30T00:59:05.213Z\n2021-05-07T19:13:19.594Z\n2021-05-07T19:13:19.594Z\n...\nHLS Sentinel-2 Multi-spectral Instrument Surfa...\n[{'URL': 'https://data.lpdaac.earthdatacloud.n...\nDay\n[{'Identifier': 'HLS.S30.T10TEK.2021127T185919...\n2021-10-29T21:47:54.000Z\n[{'Name': 'Not provided', 'SizeInBytes': 20891...\n[{'ShortName': 'Sentinel-2B', 'Instruments': [...\nhttps://cdn.earthdata.nasa.gov/umm/granule/v1.6.5\nUMM-G\n1.6.5\n\n\n3\n102.786560\ngranule\nG2144019307-LPCLOUD\n1\nHLS.L30.T10TEK.2021128T184447.v2.0\nLPCLOUD\napplication/echo10+xml\n2021-10-15T03:48:08.875Z\n2021-05-08T18:44:47.423Z\n2021-05-08T18:45:11.310Z\n...\nHLS Landsat Operational Land Imager Surface Re...\n[{'URL': 'https://data.lpdaac.earthdatacloud.n...\nDay\n[{'Identifier': 'HLS.L30.T10TEK.2021128T184447...\n2021-10-14T22:05:21.000Z\n[{'Name': 'Not provided', 'SizeInBytes': 10777...\n[{'ShortName': 'LANDSAT-8', 'Instruments': [{'...\nhttps://cdn.earthdata.nasa.gov/umm/granule/v1.6.5\nUMM-G\n1.6.5\n\n\n4\n186.955010\ngranule\nG2153591767-LPCLOUD\n1\nHLS.S30.T10TEK.2021129T184921.v2.0\nLPCLOUD\napplication/echo10+xml\n2021-10-29T16:27:32.621Z\n2021-05-09T19:03:25.151Z\n2021-05-09T19:03:25.151Z\n...\nHLS Sentinel-2 Multi-spectral Instrument Surfa...\n[{'URL': 'https://data.lpdaac.earthdatacloud.n...\nDay\n[{'Identifier': 'HLS.S30.T10TEK.2021129T184921...\n2021-10-29T16:13:58.000Z\n[{'Name': 'Not provided', 'SizeInBytes': 19603...\n[{'ShortName': 'Sentinel-2A', 'Instruments': [...\nhttps://cdn.earthdata.nasa.gov/umm/granule/v1.6.5\nUMM-G\n1.6.5\n\n\n\n\n5 rows × 24 columns\n\n\n\nWe can also preview each individual results by selecting it from the list. This will show the data links, and a browse image.\n\nresults[0]\n\n\n\n \n \n \n \n \n \n \n Data: HLS.S30.T10TEK.2021122T185911.v2.0.B10.tifHLS.S30.T10TEK.2021122T185911.v2.0.SZA.tifHLS.S30.T10TEK.2021122T185911.v2.0.B05.tifHLS.S30.T10TEK.2021122T185911.v2.0.VAA.tifHLS.S30.T10TEK.2021122T185911.v2.0.B04.tifHLS.S30.T10TEK.2021122T185911.v2.0.B01.tifHLS.S30.T10TEK.2021122T185911.v2.0.B07.tifHLS.S30.T10TEK.2021122T185911.v2.0.B02.tifHLS.S30.T10TEK.2021122T185911.v2.0.B08.tifHLS.S30.T10TEK.2021122T185911.v2.0.VZA.tifHLS.S30.T10TEK.2021122T185911.v2.0.B03.tifHLS.S30.T10TEK.2021122T185911.v2.0.B09.tifHLS.S30.T10TEK.2021122T185911.v2.0.B11.tifHLS.S30.T10TEK.2021122T185911.v2.0.B06.tifHLS.S30.T10TEK.2021122T185911.v2.0.B8A.tifHLS.S30.T10TEK.2021122T185911.v2.0.Fmask.tifHLS.S30.T10TEK.2021122T185911.v2.0.SAA.tifHLS.S30.T10TEK.2021122T185911.v2.0.B12.tif\n Size: 198.88 MB\n Cloud Hosted: True\n \n \n \n \n \n \n \n \n\n\nWe can grab all of the URLs for the data using list comprehension.\n\nhls_results_urls = [granule.data_links() for granule in results]\nprint(f\"There are {len(hls_results_urls)} granules and each granule contains {len(hls_results_urls[0])} assets. Below is an example of the assets printed for one granule:\")\npprint(hls_results_urls[0])\n\nThere are 78 granules and each granule contains 18 assets. Below is an example of the assets printed for one granule:\n['https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B10.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.SZA.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B05.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.VAA.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B01.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B07.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B02.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B08.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.VZA.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B03.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B09.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B11.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B06.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B8A.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.Fmask.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.SAA.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B12.tif']\n\n\nWe can get the URLs for the browse images as well.\n\nbrowse_urls = [granule.dataviz_links()[0] for granule in results] # 0 retrieves only the https links\npprint(browse_urls[0:5]) # Print the first five results\n\n['https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.jpg',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSS30.020/HLS.S30.T10TEK.2021124T184919.v2.0/HLS.S30.T10TEK.2021124T184919.v2.0.jpg',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSS30.020/HLS.S30.T10TEK.2021127T185919.v2.0/HLS.S30.T10TEK.2021127T185919.v2.0.jpg',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSL30.020/HLS.L30.T10TEK.2021128T184447.v2.0/HLS.L30.T10TEK.2021128T184447.v2.0.jpg',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSS30.020/HLS.S30.T10TEK.2021129T184921.v2.0/HLS.S30.T10TEK.2021129T184921.v2.0.jpg']" }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#learning-objectives", - "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#learning-objectives", - "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", - "section": "Learning Objectives", - "text": "Learning Objectives\n\nhow to retrieve temporary S3 credentials for in-region direct S3 bucket access\nhow to perform in-region direct access of ECCO_L4_SSH_05DEG_MONTHLY_V4R4 data in S3\nhow to plot the data" + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#accessing-hls-cloud-optimized-geotiffs-cogs-from-earthdata-cloud", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#accessing-hls-cloud-optimized-geotiffs-cogs-from-earthdata-cloud", + "title": "Observing Seasonality in Agricultural Areas", + "section": "3. Accessing HLS Cloud Optimized GeoTIFFs (COGs) from Earthdata Cloud ", + "text": "3. Accessing HLS Cloud Optimized GeoTIFFs (COGs) from Earthdata Cloud \nNow that we have a list of data URLs, we will configure gdal and rioxarray to access the cloud assets that we are interested in, and read them directly into memory without needing to download the files.\nThe Python libraries used to access COG files in Earthdata Cloud leverage GDAL’s virtual file systems. Whether you are running this code in the Cloud or in a local workspace, GDAL configurations must be set in order to successfully access the HLS COG files.\n\n# GDAL configurations used to successfully access LP DAAC Cloud Assets via vsicurl \ngdal.SetConfigOption('GDAL_HTTP_COOKIEFILE','~/cookies.txt')\ngdal.SetConfigOption('GDAL_HTTP_COOKIEJAR', '~/cookies.txt')\ngdal.SetConfigOption('GDAL_DISABLE_READDIR_ON_OPEN','EMPTY_DIR')\ngdal.SetConfigOption('CPL_VSIL_CURL_ALLOWED_EXTENSIONS','TIF')\ngdal.SetConfigOption('GDAL_HTTP_UNSAFESSL', 'YES')" }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#import-packages", - "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#import-packages", - "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", - "section": "Import Packages", - "text": "Import Packages\n\n%matplotlib inline\nimport matplotlib.pyplot as plt\nimport os\nimport requests\nimport s3fs\nfrom osgeo import gdal\nimport xarray as xr\nimport hvplot.xarray\nimport holoviews as hv" + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#subset-by-band", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#subset-by-band", + "title": "Observing Seasonality in Agricultural Areas", + "section": "3.1 Subset by Band", + "text": "3.1 Subset by Band\nLets take a look at the URLs for one of our returned granules.\n\nh = hls_results_urls[10] \nh\n\n['https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B12.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B10.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B01.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.Fmask.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B8A.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B09.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B11.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.VAA.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B08.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.SZA.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B07.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B06.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B02.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.VZA.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B03.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B05.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.SAA.tif']\n\n\nTo calculate the EVI for each granule we need the near-infrared, red, and blue bands. Below you can find the different band numbers for each of the two products.\n\nSentinel 2:\n - \"narrow\" NIR = B8A\n - Red = B04\n - Blue = B02 \n - Quality = Fmask\n\n\nLandsat 8:\n - NIR = B05\n - Red = B04\n - Blue = B02 \n - Quality = Fmask\nWe will subset our URLs to include the bands identified above in addition to the quality layer (Fmask).\n\nevi_band_links = []\n\n# Define which HLS product is being accessed\nif h[0].split('/')[4] == 'HLSS30.020':\n evi_bands = ['B8A', 'B04', 'B02', 'Fmask'] # NIR RED BLUE Quality for S30\nelse:\n evi_bands = ['B05', 'B04', 'B02', 'Fmask'] # NIR RED BLUE Quality for L30\n\n# Subset the assets in the item down to only the desired bands\nfor a in h: \n if any(b in a for b in evi_bands):\n evi_band_links.append(a)\nevi_band_links\n\n['https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.Fmask.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B8A.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B02.tif']\n\n\nRemember from above that you can always quickly load in the browse image to get a quick view of the item using our list of browse URLs.\n\nimage = io.imread(browse_urls[10]) # Load jpg browse image into memory\n\n# Basic plot of the image\nplt.figure(figsize=(10,10)) \nplt.imshow(image)\nplt.show()\n\n\n\n\nAbove, we see our first observation over the northern Central Valley of California.\n\ndel image # Remove the browse image" }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#get-temporary-aws-credentials", - "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#get-temporary-aws-credentials", - "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", - "section": "Get Temporary AWS Credentials", - "text": "Get Temporary AWS Credentials\nDirect S3 access is achieved by passing NASA supplied temporary credentials to AWS so we can interact with S3 objects from applicable Earthdata Cloud buckets. For now, each NASA DAAC has different AWS credentials endpoints. Below are some of the credential endpoints to various DAACs:\n\ns3_cred_endpoint = {\n 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n 'gesdisc': 'https://data.gesdisc.earthdata.nasa.gov/s3credentials',\n 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n 'ornldaac': 'https://data.ornldaac.earthdata.nasa.gov/s3credentials',\n 'ghrcdaac': 'https://data.ghrc.earthdata.nasa.gov/s3credentials'\n}\n\nCreate a function to make a request to an endpoint for temporary credentials. Remember, each DAAC has their own endpoint and credentials are not usable for cloud data from other DAACs.\n\ndef get_temp_creds(provider):\n return requests.get(s3_cred_endpoint[provider]).json()\n\n\ntemp_creds_req = get_temp_creds('podaac')\n#temp_creds_req" + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#load-hls-cogs-into-memory", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#load-hls-cogs-into-memory", + "title": "Observing Seasonality in Agricultural Areas", + "section": "3.2 Load HLS COGs into Memory", + "text": "3.2 Load HLS COGs into Memory\nHLS COGs are broken into chunks allowing data to be read more efficiently. Define the chunk size of an HLS tile, mask the NaN values, then read the files using rioxarray and name them based upon the band. We also squeeze the object to remove the band dimension from most of the files, since there is only 1 band.\n\nNOTE: To scale the bands, you can set the mask_and_scale to True (mask_and_scale=True), however the scale_factor information in some of the HLSL30 granules are found in the file metadata, but missing from the Band metadata. rioxarray looks for the scale_factor under Band metadata and if this information is missing it assumes the scale_factor is equal to 1. This results in having data to be uscaled and not masked for those granules. That is why we treat our data a bit differently here, leaving it unscaled and manually updating the scale_factor attribute in the xarray dataarray.\n\n\n# Use vsicurl to load the data directly into memory (be patient, may take a few seconds)\nchunk_size = dict(band=1, x=512, y=512) # Tiles have 1 band and are divided into 512x512 pixel chunks\nfor e in evi_band_links:\n print(e)\n if e.rsplit('.', 2)[-2] == evi_bands[0]: # NIR index\n nir = rxr.open_rasterio(e, chunks=chunk_size, masked=True).squeeze('band', drop=True)\n nir.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute \n elif e.rsplit('.', 2)[-2] == evi_bands[1]: # red index\n red = rxr.open_rasterio(e, chunks=chunk_size, masked=True).squeeze('band', drop=True)\n red.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute\n elif e.rsplit('.', 2)[-2] == evi_bands[2]: # blue index\n blue = rxr.open_rasterio(e, chunks=chunk_size, masked=True).squeeze('band', drop=True)\n blue.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute\n # elif e.rsplit('.', 2)[-2] == evi_bands[3]: # Fmask index\n # fmask = rxr.open_rasterio(e, chunks=chunk_size, masked=True).astype('uint16').squeeze('band', drop=True)\nprint(\"The COGs have been loaded into memory!\")\n\nhttps://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.Fmask.tif\nhttps://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B04.tif\nhttps://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B8A.tif\nhttps://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B02.tif\nThe COGs have been loaded into memory!\n\n\n\nrxr.open_rasterio('https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.Fmask.tif')\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (band: 1, y: 3660, x: 3660)>\n[13395600 values with dtype=uint8]\nCoordinates:\n * band (band) int32 1\n * x (x) float64 5e+05 5e+05 5.001e+05 ... 6.097e+05 6.098e+05\n * y (y) float64 4.5e+06 4.5e+06 4.5e+06 ... 4.39e+06 4.39e+06\n spatial_ref int32 0\nAttributes: (12/42)\n ACCODE: LaSRC\n AREA_OR_POINT: Area\n arop_ave_xshift(meters): 0\n arop_ave_yshift(meters): 0\n arop_ncp: 0\n arop_rmse(meters): 0\n ... ...\n TILE_ID: S2A_OPER_MSI_L1C_TL_VG...\n ULX: 499980\n ULY: 4500000\n _FillValue: 255\n scale_factor: 1.0\n add_offset: 0.0xarray.DataArrayband: 1y: 3660x: 3660...[13395600 values with dtype=uint8]Coordinates: (4)band(band)int321array([1])x(x)float645e+05 5e+05 ... 6.097e+05 6.098e+05array([499995., 500025., 500055., ..., 609705., 609735., 609765.])y(y)float644.5e+06 4.5e+06 ... 4.39e+06array([4499985., 4499955., 4499925., ..., 4390275., 4390245., 4390215.])spatial_ref()int320crs_wkt :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not specified (based on WGS 84 spheroid)projected_crs_name :UTM Zone 10, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-123.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :499980.0 30.0 0.0 4500000.0 0.0 -30.0array(0)Indexes: (3)bandPandasIndexPandasIndex(Index([1], dtype='int32', name='band'))xPandasIndexPandasIndex(Index([499995.0, 500025.0, 500055.0, 500085.0, 500115.0, 500145.0, 500175.0,\n 500205.0, 500235.0, 500265.0,\n ...\n 609495.0, 609525.0, 609555.0, 609585.0, 609615.0, 609645.0, 609675.0,\n 609705.0, 609735.0, 609765.0],\n dtype='float64', name='x', length=3660))yPandasIndexPandasIndex(Index([4499985.0, 4499955.0, 4499925.0, 4499895.0, 4499865.0, 4499835.0,\n 4499805.0, 4499775.0, 4499745.0, 4499715.0,\n ...\n 4390485.0, 4390455.0, 4390425.0, 4390395.0, 4390365.0, 4390335.0,\n 4390305.0, 4390275.0, 4390245.0, 4390215.0],\n dtype='float64', name='y', length=3660))Attributes: (42)ACCODE :LaSRCAREA_OR_POINT :Areaarop_ave_xshift(meters) :0arop_ave_yshift(meters) :0arop_ncp :0arop_rmse(meters) :0arop_s2_refimg :NONEcloud_coverage :31DATASTRIP_ID :S2A_OPER_MSI_L1C_DS_VGS4_20210523T002123_S20210522T191158_N03.00Fmask description :Bits are listed from the MSB (bit 7) to the LSB (bit 0): \n7-6 aerosol:\n 00 - climatology\n 01 - low\n 10 - average\n 11 - high\n5 water\n4 snow/ice\n3 cloud shadow\n2 adjacent to cloud\n1 cloud\n0 cirrus cloud\nHLS_PROCESSING_TIME :2021-10-24T00:46:20ZHORIZONTAL_CS_CODE :EPSG:32610HORIZONTAL_CS_NAME :WGS84 / UTM zone 10NL1C_IMAGE_QUALITY :NONEL1_PROCESSING_TIME :2021-05-23T00:58:53.432141ZMEAN_SUN_AZIMUTH_ANGLE :146.1715126044MEAN_SUN_ZENITH_ANGLE :22.6352110476638MEAN_VIEW_AZIMUTH_ANGLE :284.987205692531MEAN_VIEW_ZENITH_ANGLE :7.8847569388168MSI band 01 bandpass adjustment slope and offset :0.995900, -0.000200MSI band 02 bandpass adjustment slope and offset :0.977800, -0.004000MSI band 03 bandpass adjustment slope and offset :1.005300, -0.000900MSI band 04 bandpass adjustment slope and offset :0.976500, 0.000900MSI band 11 bandpass adjustment slope and offset :0.998700, -0.001100MSI band 12 bandpass adjustment slope and offset :1.003000, -0.001200MSI band 8a bandpass adjustment slope and offset :0.998300, -0.000100NBAR_SOLAR_ZENITH :24.4031571183093NCOLS :3660NROWS :3660OVR_RESAMPLING_ALG :NEARESTPROCESSING_BASELINE :3.0PRODUCT_URI :S2A_MSIL1C_20210522T185921_N0300_R013_T10TEK_20210523T002123.SAFESENSING_TIME :2021-05-22T19:13:22.446684ZSPACECRAFT_NAME :Sentinel-2Aspatial_coverage :92SPATIAL_RESOLUTION :30TILE_ID :S2A_OPER_MSI_L1C_TL_VGS4_20210523T002123_A030898_T10TEK_N03.00ULX :499980ULY :4500000_FillValue :255scale_factor :1.0add_offset :0.0\n\n\n\nNOTE: Getting an error in the section above? Accessing these files in the cloud requires you to authenticate using your NASA Earthdata Login account. You will need to have a netrc file set up containing those credentials in your home directory in order to successfully run the code below. Please make sure you have a valid username and password in the created netrc file.\n\nWe can take a quick look at one of the dataarray we just read in.\n\nnir\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (y: 3660, x: 3660)>\ndask.array<getitem, shape=(3660, 3660), dtype=float32, chunksize=(512, 512), chunktype=numpy.ndarray>\nCoordinates:\n * x (x) float64 5e+05 5e+05 5.001e+05 ... 6.097e+05 6.098e+05\n * y (y) float64 4.5e+06 4.5e+06 4.5e+06 ... 4.39e+06 4.39e+06\n spatial_ref int32 0\nAttributes: (12/41)\n ACCODE: LaSRC\n add_offset: 0.0\n AREA_OR_POINT: Area\n arop_ave_xshift(meters): 0\n arop_ave_yshift(meters): 0\n arop_ncp: 0\n ... ...\n SPACECRAFT_NAME: Sentinel-2A\n spatial_coverage: 92\n SPATIAL_RESOLUTION: 30\n TILE_ID: S2A_OPER_MSI_L1C_TL_VG...\n ULX: 499980\n ULY: 4500000xarray.DataArrayy: 3660x: 3660dask.array<chunksize=(512, 512), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n51.10 MiB\n1.00 MiB\n\n\nShape\n(3660, 3660)\n(512, 512)\n\n\nDask graph\n64 chunks in 3 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\nCoordinates: (3)x(x)float645e+05 5e+05 ... 6.097e+05 6.098e+05array([499995., 500025., 500055., ..., 609705., 609735., 609765.])y(y)float644.5e+06 4.5e+06 ... 4.39e+06array([4499985., 4499955., 4499925., ..., 4390275., 4390245., 4390215.])spatial_ref()int320crs_wkt :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not specified (based on WGS 84 spheroid)projected_crs_name :UTM Zone 10, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-123.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :499980.0 30.0 0.0 4500000.0 0.0 -30.0array(0)Indexes: (2)xPandasIndexPandasIndex(Index([499995.0, 500025.0, 500055.0, 500085.0, 500115.0, 500145.0, 500175.0,\n 500205.0, 500235.0, 500265.0,\n ...\n 609495.0, 609525.0, 609555.0, 609585.0, 609615.0, 609645.0, 609675.0,\n 609705.0, 609735.0, 609765.0],\n dtype='float64', name='x', length=3660))yPandasIndexPandasIndex(Index([4499985.0, 4499955.0, 4499925.0, 4499895.0, 4499865.0, 4499835.0,\n 4499805.0, 4499775.0, 4499745.0, 4499715.0,\n ...\n 4390485.0, 4390455.0, 4390425.0, 4390395.0, 4390365.0, 4390335.0,\n 4390305.0, 4390275.0, 4390245.0, 4390215.0],\n dtype='float64', name='y', length=3660))Attributes: (41)ACCODE :LaSRCadd_offset :0.0AREA_OR_POINT :Areaarop_ave_xshift(meters) :0arop_ave_yshift(meters) :0arop_ncp :0arop_rmse(meters) :0arop_s2_refimg :NONEcloud_coverage :31DATASTRIP_ID :S2A_OPER_MSI_L1C_DS_VGS4_20210523T002123_S20210522T191158_N03.00HLS_PROCESSING_TIME :2021-10-24T00:46:20ZHORIZONTAL_CS_CODE :EPSG:32610HORIZONTAL_CS_NAME :WGS84 / UTM zone 10NL1C_IMAGE_QUALITY :NONEL1_PROCESSING_TIME :2021-05-23T00:58:53.432141Zlong_name :NIR_NarrowMEAN_SUN_AZIMUTH_ANGLE :146.1715126044MEAN_SUN_ZENITH_ANGLE :22.6352110476638MEAN_VIEW_AZIMUTH_ANGLE :284.987205692531MEAN_VIEW_ZENITH_ANGLE :7.8847569388168MSI band 01 bandpass adjustment slope and offset :0.995900, -0.000200MSI band 02 bandpass adjustment slope and offset :0.977800, -0.004000MSI band 03 bandpass adjustment slope and offset :1.005300, -0.000900MSI band 04 bandpass adjustment slope and offset :0.976500, 0.000900MSI band 11 bandpass adjustment slope and offset :0.998700, -0.001100MSI band 12 bandpass adjustment slope and offset :1.003000, -0.001200MSI band 8a bandpass adjustment slope and offset :0.998300, -0.000100NBAR_SOLAR_ZENITH :24.4031571183093NCOLS :3660NROWS :3660OVR_RESAMPLING_ALG :NEARESTPROCESSING_BASELINE :3.0PRODUCT_URI :S2A_MSIL1C_20210522T185921_N0300_R013_T10TEK_20210523T002123.SAFEscale_factor :0.0001SENSING_TIME :2021-05-22T19:13:22.446684ZSPACECRAFT_NAME :Sentinel-2Aspatial_coverage :92SPATIAL_RESOLUTION :30TILE_ID :S2A_OPER_MSI_L1C_TL_VGS4_20210523T002123_A030898_T10TEK_N03.00ULX :499980ULY :4500000\n\n\nNote the full size of the array, y=3660 & x=3660" }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#set-up-an-s3fs-session-for-direct-access", - "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#set-up-an-s3fs-session-for-direct-access", - "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", - "section": "Set up an s3fs session for Direct Access", - "text": "Set up an s3fs session for Direct Access\ns3fs sessions are used for authenticated access to s3 bucket and allows for typical file-system style operations. Below we create session by passing in the temporary credentials we recieved from our temporary credentials endpoint.\n\nfs_s3 = s3fs.S3FileSystem(anon=False, \n key=temp_creds_req['accessKeyId'], \n secret=temp_creds_req['secretAccessKey'], \n token=temp_creds_req['sessionToken'])\n\nIn this example we’re interested in the ECCO data collection from NASA’s PO.DAAC in Earthdata Cloud. Below we specify the s3 URL to the data asset in Earthdata Cloud. This URL can be found via Earthdata Search or programmatically through the CMR and CMR-STAC APIs.\n\ns3_url = 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-01_ECCO_V4r4_latlon_0p50deg.nc'" + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#subset-spatially", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#subset-spatially", + "title": "Observing Seasonality in Agricultural Areas", + "section": "3.3 Subset spatially", + "text": "3.3 Subset spatially\nBefore we can subset using our input farm field, we will first need to convert the geopandas dataframe from lat/lon (EPSG: 4326) into the projection used by HLS, UTM (aligned to the Military Grid Reference System). Since UTM is a zonal projection, we’ll need to extract the unique UTM zonal projection parameters from our input HLS files and use them to transform the coordinate of our input farm field.\nWe can print out the WKT string for our HLS tiles.\n\nnir.spatial_ref.crs_wkt\n\n'PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]'\n\n\nWe will use this information to transform the coordinates of our ROI to the proper UTM projection.\n\nfsUTM = field.to_crs(nir.spatial_ref.crs_wkt) # Take the CRS from the NIR tile that we opened and apply it to our field geodataframe.\nfsUTM\n\n\n\n\n\n\n\n\ngeometry\n\n\n\n\n0\nPOLYGON ((581047.926 4418541.913, 580150.196 4...\n\n\n\n\n\n\n\nNow, we can use our field ROI to mask any pixels that fall outside of it and crop to the bounding box using rasterio. This greatly reduces the amount of data that are needed to load into memory.\n\nnir_cropped = nir.rio.clip(fsUTM.geometry.values, fsUTM.crs, all_touched=True) # All touched includes any pixels touched by the polygon\nnir_cropped\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (y: 59, x: 38)>\ndask.array<getitem, shape=(59, 38), dtype=float32, chunksize=(59, 38), chunktype=numpy.ndarray>\nCoordinates:\n * x (x) float64 5.802e+05 5.802e+05 ... 5.812e+05 5.813e+05\n * y (y) float64 4.419e+06 4.419e+06 ... 4.417e+06 4.417e+06\n spatial_ref int32 0\nAttributes: (12/41)\n ACCODE: LaSRC\n add_offset: 0.0\n AREA_OR_POINT: Area\n arop_ave_xshift(meters): 0\n arop_ave_yshift(meters): 0\n arop_ncp: 0\n ... ...\n SPACECRAFT_NAME: Sentinel-2A\n spatial_coverage: 92\n SPATIAL_RESOLUTION: 30\n TILE_ID: S2A_OPER_MSI_L1C_TL_VG...\n ULX: 499980\n ULY: 4500000xarray.DataArrayy: 59x: 38dask.array<chunksize=(59, 38), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n8.76 kiB\n8.76 kiB\n\n\nShape\n(59, 38)\n(59, 38)\n\n\nDask graph\n1 chunks in 7 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\nCoordinates: (3)x(x)float645.802e+05 5.802e+05 ... 5.813e+05axis :Xlong_name :x coordinate of projectionstandard_name :projection_x_coordinateunits :metrearray([580155., 580185., 580215., 580245., 580275., 580305., 580335., 580365.,\n 580395., 580425., 580455., 580485., 580515., 580545., 580575., 580605.,\n 580635., 580665., 580695., 580725., 580755., 580785., 580815., 580845.,\n 580875., 580905., 580935., 580965., 580995., 581025., 581055., 581085.,\n 581115., 581145., 581175., 581205., 581235., 581265.])y(y)float644.419e+06 4.419e+06 ... 4.417e+06axis :Ylong_name :y coordinate of projectionstandard_name :projection_y_coordinateunits :metrearray([4418535., 4418505., 4418475., 4418445., 4418415., 4418385., 4418355.,\n 4418325., 4418295., 4418265., 4418235., 4418205., 4418175., 4418145.,\n 4418115., 4418085., 4418055., 4418025., 4417995., 4417965., 4417935.,\n 4417905., 4417875., 4417845., 4417815., 4417785., 4417755., 4417725.,\n 4417695., 4417665., 4417635., 4417605., 4417575., 4417545., 4417515.,\n 4417485., 4417455., 4417425., 4417395., 4417365., 4417335., 4417305.,\n 4417275., 4417245., 4417215., 4417185., 4417155., 4417125., 4417095.,\n 4417065., 4417035., 4417005., 4416975., 4416945., 4416915., 4416885.,\n 4416855., 4416825., 4416795.])spatial_ref()int320crs_wkt :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not specified (based on WGS 84 spheroid)projected_crs_name :UTM Zone 10, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-123.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :580140.0 30.0 0.0 4418550.0 0.0 -30.0array(0)Indexes: (2)xPandasIndexPandasIndex(Index([580155.0, 580185.0, 580215.0, 580245.0, 580275.0, 580305.0, 580335.0,\n 580365.0, 580395.0, 580425.0, 580455.0, 580485.0, 580515.0, 580545.0,\n 580575.0, 580605.0, 580635.0, 580665.0, 580695.0, 580725.0, 580755.0,\n 580785.0, 580815.0, 580845.0, 580875.0, 580905.0, 580935.0, 580965.0,\n 580995.0, 581025.0, 581055.0, 581085.0, 581115.0, 581145.0, 581175.0,\n 581205.0, 581235.0, 581265.0],\n dtype='float64', name='x'))yPandasIndexPandasIndex(Index([4418535.0, 4418505.0, 4418475.0, 4418445.0, 4418415.0, 4418385.0,\n 4418355.0, 4418325.0, 4418295.0, 4418265.0, 4418235.0, 4418205.0,\n 4418175.0, 4418145.0, 4418115.0, 4418085.0, 4418055.0, 4418025.0,\n 4417995.0, 4417965.0, 4417935.0, 4417905.0, 4417875.0, 4417845.0,\n 4417815.0, 4417785.0, 4417755.0, 4417725.0, 4417695.0, 4417665.0,\n 4417635.0, 4417605.0, 4417575.0, 4417545.0, 4417515.0, 4417485.0,\n 4417455.0, 4417425.0, 4417395.0, 4417365.0, 4417335.0, 4417305.0,\n 4417275.0, 4417245.0, 4417215.0, 4417185.0, 4417155.0, 4417125.0,\n 4417095.0, 4417065.0, 4417035.0, 4417005.0, 4416975.0, 4416945.0,\n 4416915.0, 4416885.0, 4416855.0, 4416825.0, 4416795.0],\n dtype='float64', name='y'))Attributes: (41)ACCODE :LaSRCadd_offset :0.0AREA_OR_POINT :Areaarop_ave_xshift(meters) :0arop_ave_yshift(meters) :0arop_ncp :0arop_rmse(meters) :0arop_s2_refimg :NONEcloud_coverage :31DATASTRIP_ID :S2A_OPER_MSI_L1C_DS_VGS4_20210523T002123_S20210522T191158_N03.00HLS_PROCESSING_TIME :2021-10-24T00:46:20ZHORIZONTAL_CS_CODE :EPSG:32610HORIZONTAL_CS_NAME :WGS84 / UTM zone 10NL1C_IMAGE_QUALITY :NONEL1_PROCESSING_TIME :2021-05-23T00:58:53.432141Zlong_name :NIR_NarrowMEAN_SUN_AZIMUTH_ANGLE :146.1715126044MEAN_SUN_ZENITH_ANGLE :22.6352110476638MEAN_VIEW_AZIMUTH_ANGLE :284.987205692531MEAN_VIEW_ZENITH_ANGLE :7.8847569388168MSI band 01 bandpass adjustment slope and offset :0.995900, -0.000200MSI band 02 bandpass adjustment slope and offset :0.977800, -0.004000MSI band 03 bandpass adjustment slope and offset :1.005300, -0.000900MSI band 04 bandpass adjustment slope and offset :0.976500, 0.000900MSI band 11 bandpass adjustment slope and offset :0.998700, -0.001100MSI band 12 bandpass adjustment slope and offset :1.003000, -0.001200MSI band 8a bandpass adjustment slope and offset :0.998300, -0.000100NBAR_SOLAR_ZENITH :24.4031571183093NCOLS :3660NROWS :3660OVR_RESAMPLING_ALG :NEARESTPROCESSING_BASELINE :3.0PRODUCT_URI :S2A_MSIL1C_20210522T185921_N0300_R013_T10TEK_20210523T002123.SAFEscale_factor :0.0001SENSING_TIME :2021-05-22T19:13:22.446684ZSPACECRAFT_NAME :Sentinel-2Aspatial_coverage :92SPATIAL_RESOLUTION :30TILE_ID :S2A_OPER_MSI_L1C_TL_VGS4_20210523T002123_A030898_T10TEK_N03.00ULX :499980ULY :4500000\n\n\nNote that the array size is considerably smaller than the full size we read in before.\nNow we will plot the cropped NIR data.\n\nnir_cropped.hvplot.image(aspect='equal', cmap='viridis', frame_height=500, frame_width= 500).opts(title='HLS Cropped NIR Band') # Quick visual to assure that it worked\n\n\n\n\n\n \n\n\n\n\nAbove, you can see that the data have been loaded into memory already subset to our ROI. Also notice that the data has not been scaled (see the legend). We will next scaled the data using the function defined below." }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#direct-in-region-access", - "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#direct-in-region-access", - "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", - "section": "Direct In-region Access", - "text": "Direct In-region Access\nOpen with the netCDF file using the s3fs package, then load the cloud asset into an xarray dataset.\n\ns3_file_obj = fs_s3.open(s3_url, mode='rb')\n\n\nssh_ds = xr.open_dataset(s3_file_obj, engine='h5netcdf')\nssh_ds\n\nGet the SSH variable as an xarray dataarray\n\nssh_da = ssh_ds.SSH\nssh_da\n\nPlot the SSH dataarray for time 2015-01-16T12:00:00 using hvplot.\n\nssh_da.hvplot.image(x='longitude', y='latitude', cmap='Spectral_r', geo=True, tiles='ESRI', global_extent=True)" + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#apply-scale-factor", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#apply-scale-factor", + "title": "Observing Seasonality in Agricultural Areas", + "section": "3.4 Apply Scale Factor", + "text": "3.4 Apply Scale Factor\n\n# Define function to scale \ndef scaling(band):\n scale_factor = band.attrs['scale_factor'] \n band_out = band.copy()\n band_out.data = band.data*scale_factor\n band_out.attrs['scale_factor'] = 1\n return(band_out)\n\n\nnir_cropped_scaled = scaling(nir_cropped)\n\nWe can plot to confirm our manual scaling worked.\n\nnir_cropped_scaled.hvplot.image(aspect='equal', cmap='viridis', frame_height=500, frame_width= 500).opts(title='HLS Cropped NIR Band') # Quick visual to assure that it worked\n\n\n\n\n\n \n\n\n\n\nNext, load in the red and blue bands and fix their scaling as well.\n\n# Red\nred_cropped = red.rio.clip(fsUTM.geometry.values, fsUTM.crs, all_touched=True)\nred_cropped_scaled = scaling(red_cropped)\n# Blue\nblue_cropped = blue.rio.clip(fsUTM.geometry.values, fsUTM.crs, all_touched=True)\nblue_cropped_scaled = scaling(blue_cropped)\nprint('Data is loaded and scaled!')\n\nData is loaded and scaled!" }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#resources", - "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#resources", - "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", - "section": "Resources", - "text": "Resources\nDirect access to ECCO data in S3 (from us-west-2)\nData_Access__Direct_S3_Access__PODAAC_ECCO_SSH using CMR-STAC API to retrieve S3 links" + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#processing-hls-data", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#processing-hls-data", + "title": "Observing Seasonality in Agricultural Areas", + "section": "4. Processing HLS Data ", + "text": "4. Processing HLS Data \nIn this section we will use the HLS data we’ve access to calculate the EVI. We will do this by defining a function to calculate EVI that will retain the attributes and metadata associated with the data we accessed.\n\n4.1 Calculate EVI\nBelow is a function we will use to calculate EVI using the NIR, Red, and Blue bands. The function will build a new xarray dataarray with EVI values and copy the original file metadata to the new xarray dataarray\n\ndef calc_evi(red, blue, nir):\n # Create EVI xarray.DataArray that has the same coordinates and metadata \n evi = red.copy()\n # Calculate the EVI\n evi_data = 2.5 * ((nir.data - red.data) / (nir.data + 6.0 * red.data - 7.5 * blue.data + 1.0))\n # Replace the Red xarray.DataArray data with the new EVI data\n evi.data = evi_data\n # exclude the inf values\n evi = xr.where(evi != np.inf, evi, np.nan, keep_attrs=True)\n # change the long_name in the attributes\n evi.attrs['long_name'] = 'EVI'\n evi.attrs['scale_factor'] = 1\n return evi\n\nBelow, apply the EVI function on the scaled data.\n\nevi_cropped = calc_evi(red_cropped_scaled, blue_cropped_scaled, nir_cropped_scaled) # Generate EVI array\nevi_cropped\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (y: 59, x: 38)>\ndask.array<where, shape=(59, 38), dtype=float32, chunksize=(59, 38), chunktype=numpy.ndarray>\nCoordinates:\n * x (x) float64 5.802e+05 5.802e+05 ... 5.812e+05 5.813e+05\n * y (y) float64 4.419e+06 4.419e+06 ... 4.417e+06 4.417e+06\n spatial_ref int32 0\nAttributes: (12/41)\n ACCODE: LaSRC\n add_offset: 0.0\n AREA_OR_POINT: Area\n arop_ave_xshift(meters): 0\n arop_ave_yshift(meters): 0\n arop_ncp: 0\n ... ...\n SPACECRAFT_NAME: Sentinel-2A\n spatial_coverage: 92\n SPATIAL_RESOLUTION: 30\n TILE_ID: S2A_OPER_MSI_L1C_TL_VG...\n ULX: 499980\n ULY: 4500000xarray.DataArrayy: 59x: 38dask.array<chunksize=(59, 38), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n8.76 kiB\n8.76 kiB\n\n\nShape\n(59, 38)\n(59, 38)\n\n\nDask graph\n1 chunks in 30 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\nCoordinates: (3)x(x)float645.802e+05 5.802e+05 ... 5.813e+05axis :Xlong_name :x coordinate of projectionstandard_name :projection_x_coordinateunits :metrearray([580155., 580185., 580215., 580245., 580275., 580305., 580335., 580365.,\n 580395., 580425., 580455., 580485., 580515., 580545., 580575., 580605.,\n 580635., 580665., 580695., 580725., 580755., 580785., 580815., 580845.,\n 580875., 580905., 580935., 580965., 580995., 581025., 581055., 581085.,\n 581115., 581145., 581175., 581205., 581235., 581265.])y(y)float644.419e+06 4.419e+06 ... 4.417e+06axis :Ylong_name :y coordinate of projectionstandard_name :projection_y_coordinateunits :metrearray([4418535., 4418505., 4418475., 4418445., 4418415., 4418385., 4418355.,\n 4418325., 4418295., 4418265., 4418235., 4418205., 4418175., 4418145.,\n 4418115., 4418085., 4418055., 4418025., 4417995., 4417965., 4417935.,\n 4417905., 4417875., 4417845., 4417815., 4417785., 4417755., 4417725.,\n 4417695., 4417665., 4417635., 4417605., 4417575., 4417545., 4417515.,\n 4417485., 4417455., 4417425., 4417395., 4417365., 4417335., 4417305.,\n 4417275., 4417245., 4417215., 4417185., 4417155., 4417125., 4417095.,\n 4417065., 4417035., 4417005., 4416975., 4416945., 4416915., 4416885.,\n 4416855., 4416825., 4416795.])spatial_ref()int320crs_wkt :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not specified (based on WGS 84 spheroid)projected_crs_name :UTM Zone 10, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-123.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :580140.0 30.0 0.0 4418550.0 0.0 -30.0array(0)Indexes: (2)xPandasIndexPandasIndex(Index([580155.0, 580185.0, 580215.0, 580245.0, 580275.0, 580305.0, 580335.0,\n 580365.0, 580395.0, 580425.0, 580455.0, 580485.0, 580515.0, 580545.0,\n 580575.0, 580605.0, 580635.0, 580665.0, 580695.0, 580725.0, 580755.0,\n 580785.0, 580815.0, 580845.0, 580875.0, 580905.0, 580935.0, 580965.0,\n 580995.0, 581025.0, 581055.0, 581085.0, 581115.0, 581145.0, 581175.0,\n 581205.0, 581235.0, 581265.0],\n dtype='float64', name='x'))yPandasIndexPandasIndex(Index([4418535.0, 4418505.0, 4418475.0, 4418445.0, 4418415.0, 4418385.0,\n 4418355.0, 4418325.0, 4418295.0, 4418265.0, 4418235.0, 4418205.0,\n 4418175.0, 4418145.0, 4418115.0, 4418085.0, 4418055.0, 4418025.0,\n 4417995.0, 4417965.0, 4417935.0, 4417905.0, 4417875.0, 4417845.0,\n 4417815.0, 4417785.0, 4417755.0, 4417725.0, 4417695.0, 4417665.0,\n 4417635.0, 4417605.0, 4417575.0, 4417545.0, 4417515.0, 4417485.0,\n 4417455.0, 4417425.0, 4417395.0, 4417365.0, 4417335.0, 4417305.0,\n 4417275.0, 4417245.0, 4417215.0, 4417185.0, 4417155.0, 4417125.0,\n 4417095.0, 4417065.0, 4417035.0, 4417005.0, 4416975.0, 4416945.0,\n 4416915.0, 4416885.0, 4416855.0, 4416825.0, 4416795.0],\n dtype='float64', name='y'))Attributes: (41)ACCODE :LaSRCadd_offset :0.0AREA_OR_POINT :Areaarop_ave_xshift(meters) :0arop_ave_yshift(meters) :0arop_ncp :0arop_rmse(meters) :0arop_s2_refimg :NONEcloud_coverage :31DATASTRIP_ID :S2A_OPER_MSI_L1C_DS_VGS4_20210523T002123_S20210522T191158_N03.00HLS_PROCESSING_TIME :2021-10-24T00:46:20ZHORIZONTAL_CS_CODE :EPSG:32610HORIZONTAL_CS_NAME :WGS84 / UTM zone 10NL1C_IMAGE_QUALITY :NONEL1_PROCESSING_TIME :2021-05-23T00:58:53.432141Zlong_name :EVIMEAN_SUN_AZIMUTH_ANGLE :146.1715126044MEAN_SUN_ZENITH_ANGLE :22.6352110476638MEAN_VIEW_AZIMUTH_ANGLE :284.987205692531MEAN_VIEW_ZENITH_ANGLE :7.8847569388168MSI band 01 bandpass adjustment slope and offset :0.995900, -0.000200MSI band 02 bandpass adjustment slope and offset :0.977800, -0.004000MSI band 03 bandpass adjustment slope and offset :1.005300, -0.000900MSI band 04 bandpass adjustment slope and offset :0.976500, 0.000900MSI band 11 bandpass adjustment slope and offset :0.998700, -0.001100MSI band 12 bandpass adjustment slope and offset :1.003000, -0.001200MSI band 8a bandpass adjustment slope and offset :0.998300, -0.000100NBAR_SOLAR_ZENITH :24.4031571183093NCOLS :3660NROWS :3660OVR_RESAMPLING_ALG :NEARESTPROCESSING_BASELINE :3.0PRODUCT_URI :S2A_MSIL1C_20210522T185921_N0300_R013_T10TEK_20210523T002123.SAFEscale_factor :1SENSING_TIME :2021-05-22T19:13:22.446684ZSPACECRAFT_NAME :Sentinel-2Aspatial_coverage :92SPATIAL_RESOLUTION :30TILE_ID :S2A_OPER_MSI_L1C_TL_VGS4_20210523T002123_A030898_T10TEK_N03.00ULX :499980ULY :4500000\n\n\nNext, plot the results using hvplot.\n\nevi_cropped.hvplot.image(aspect='equal', cmap='YlGn', frame_height=500, frame_width= 500).opts(title=f'HLS-derived EVI, {evi_cropped.SENSING_TIME}', clabel='EVI')\n\n\n\n\n\n \n\n\n\n\nAbove, notice that variation of green level appearing in different fields in our ROI, some being much greener than the others.\n\n\n4.2. Export to COG\nIn this section, create an output filename and export the cropped EVI to COG.\n\noriginal_name = evi_band_links[0].split('/')[-1]\noriginal_name\n\n'HLS.S30.T10TEK.2021142T185921.v2.0.Fmask.tif'\n\n\nThe standard format for HLS S30 V2.0 and HLS L30 V2.0 filenames is as follows: > HLS.S30/HLS.L30: Product Short Name\nT10TEK: MGRS Tile ID (T+5-digits)\n2020273T190109: Julian Date and Time of Acquisition (YYYYDDDTHHMMSS)\nv2.0: Product Version\nB8A/B05: Spectral Band\n.tif: Data Format (Cloud Optimized GeoTIFF)\nFor additional information on HLS naming conventions, be sure to check out the HLS Overview Page.\nNow modify the filename to describe that its EVI, cropped to an ROI.\n\nout_name = f\"{original_name.split('v2.0')[0]}v2.0_EVI_cropped.tif\" # Generate output name from the original filename\nout_name\n\n'HLS.S30.T10TEK.2021142T185921.v2.0_EVI_cropped.tif'\n\n\nUse the COG driver to write a local raster output. A cloud-optimized geotiff (COG) is a geotiff file that has been tiled and includes overviews so it can be accessed and previewed without having to load the entire image into memory at once.\n\nout_folder = './data/'\nif not os.path.exists(out_folder): os.makedirs(out_folder)\n\nevi_cropped.rio.to_raster(raster_path=f'{out_folder}{out_name}', driver='COG')\n\n\n# del evi_cropped, out_folder, out_name, fmask_cropped, red_cropped, blue_cropped, nir_cropped, red_cropped_scaled, blue_cropped_scaled, nir_cropped_scaled" }, { - "objectID": "appendix/authentication.html", - "href": "appendix/authentication.html", - "title": "Authentication", - "section": "", - "text": "An Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\nWe recommend authentication using earthaccess Python package, which you can also call from R. Please check the how do I find data using code on how to utilize the earthaccess package.\nAuthentication is also possible without the earthaccess package. You will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. The following code creates a hidden .netrc file (_netrc for Windows OS) with Earthdata login credentials in your home directory. This file is needed to access NASA Earthdata assets from scripting environments like Python and R. Below is code describing this process.\n\nPythonR\n\n\nPlease see this 2021 Cloud Hackathon tutorial for Python code.\n\n\nFor authentication in R, please run the code below:\n\n\n\n[R code]\n\n# Required packages for this script\npackages = c('sys', 'getPass')\n\n# Identify missing (not installed) packages\nnew.packages = packages[!(packages %in% installed.packages()[,\"Package\"])]\n\n# Install missing packages\nif(length(new.packages)) install.packages(new.packages, repos='http://cran.rstudio.com/')\n\n# Load packages into R\nlibrary(sys)\nlibrary(getPass)\n\n# Determine OS and associated netrc file \nnetrc_type <- if(.Platform$OS.type == \"windows\") \"_netrc\" else \".netrc\" # Windows OS uses _netrc file\n\n# Specify path to user profile \nup <- file.path(Sys.getenv(\"USERPROFILE\")) # Retrieve user directory (for netrc file)\n\n# Below, HOME and Userprofile directories are set. \n\nif (up == \"\") {\n up <- Sys.getenv(\"HOME\") \n Sys.setenv(\"userprofile\" = up)\n if (up == \"\") {\n cat('USERPROFILE/HOME directories need to be set up. Please type sys.setenv(\"HOME\" = \"YOURDIRECTORY\") or sys.setenv(\"USERPROFILE\" = \"YOURDIRECTORY\") in your console and type your USERPROFILE/HOME direcory instead of \"YOURDIRECTORY\". Next, run the code chunk again.')\n }\n} else {Sys.setenv(\"HOME\" = up)} \n\nnetrc_path <- file.path(up, netrc_type, fsep = .Platform$file.sep) # Path to netrc file\n\n# Create a netrc file if one does not exist already\nif (file.exists(netrc_path) == FALSE || grepl(\"urs.earthdata.nasa.gov\", readLines(netrc_path)) == FALSE) {\n netrc_conn <- file(netrc_path)\n \n # User will be prompted for NASA Earthdata Login Username and Password below\n writeLines(c(\"machine urs.earthdata.nasa.gov\",\n sprintf(\"login %s\", getPass(msg = \"Enter NASA Earthdata Login Username \\n (An account can be Created at urs.earthdata.nasa.gov):\")),\n sprintf(\"password %s\", getPass(msg = \"Enter NASA Earthdata Login Password:\"))), netrc_conn)\n close(netrc_conn)\n}else{\n i <- 0 \n for (f in readLines(netrc_path)){\n i <- i + 1\n if (f ==\"machine urs.earthdata.nasa.gov\"){\n username <- strsplit(readLines(netrc_path)[i+1], \" \")[[1]][2]\n un <- getPass(msg = paste0(\"Is your NASA Earthdata Login Username: \", username, \"\\n\\n Type yes or no.\"))\n if (tolower(un) == 'yes'){\n tx <- gsub(readLines(netrc_path)[i+2], sprintf(\"password %s\", getPass(msg = \"Enter NASA Earthdata Login Password:\")), readLines(netrc_path))\n writeLines(tx, netrc_path)\n rm(username, un, tx, f, i)\n }else{\n user <- gsub(readLines(netrc_path)[i+1], sprintf(\"login %s\", getPass(msg = \"Enter NASA Earthdata Login Username:\")), readLines(netrc_path))\n tx <- gsub(readLines(netrc_path)[i+2], sprintf(\"password %s\", getPass(msg = \"Enter NASA Earthdata Login Password:\")), readLines(netrc_path))\n writeLines(tx, netrc_path)\n rm(username, un, user, tx, f, i)\n \n }\n break\n }\n }\n}" + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#automation", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#automation", + "title": "Observing Seasonality in Agricultural Areas", + "section": "5. Automation ", + "text": "5. Automation \nIn this section, automate sections 4-5 for each HLS item that intersects our spatiotemporal subset of interest. Loop through each item and subset to the desired bands, load the spatial subset into memory, apply the scale factor, calculate EVI, and export as a Cloud Optimized GeoTIFF.\n\nlen(hls_results_urls)\n\n78\n\n\nNow put it all together and loop through each of the files, visualize and export cropped EVI files. Be patient with the for loop below, it may take a few minutes to complete.\n\nfor j, h in enumerate(hls_results_urls):\n \n outName = h[0].split('/')[-1].split('v2.0')[0] +'v2.0_EVI_cropped.tif'\n #print(outName)\n\n evi_band_links = []\n if h[0].split('/')[4] == 'HLSS30.020':\n evi_bands = ['B8A', 'B04', 'B02', 'Fmask'] # NIR RED BLUE FMASK\n else:\n evi_bands = ['B05', 'B04', 'B02', 'Fmask'] # NIR RED BLUE FMASK\n \n for a in h: \n if any(b in a for b in evi_bands):\n evi_band_links.append(a)\n\n \n # Check if file already exists in output directory, if yes--skip that file and move to the next observation\n if os.path.exists(f'{out_folder}{outName}'):\n print(f\"{outName} has already been processed and is available in this directory, moving to next file.\")\n continue\n \n # Use vsicurl to load the data directly into memory (be patient, may take a few seconds)\n chunk_size = dict(band=1, x=512, y=512) # Tiles have 1 band and are divided into 512x512 pixel chunks\n for e in evi_band_links:\n #print(e)\n if e.rsplit('.', 2)[-2] == evi_bands[0]: # NIR index\n nir = rxr.open_rasterio(e, chunks=chunk_size, masked= True).squeeze('band', drop=True)\n nir.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute \n elif e.rsplit('.', 2)[-2] == evi_bands[1]: # red index\n red = rxr.open_rasterio(e, chunks=chunk_size, masked= True).squeeze('band', drop=True)\n red.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute\n elif e.rsplit('.', 2)[-2] == evi_bands[2]: # blue index\n blue = rxr.open_rasterio(e, chunks=chunk_size, masked= True).squeeze('band', drop=True)\n blue.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute\n elif e.rsplit('.', 2)[-2] == evi_bands[3]: # Fmask index\n fmask = rxr.open_rasterio(e, chunks=chunk_size, masked= True).astype('uint16').squeeze('band', drop=True)\n\n \n #print(\"The COGs have been loaded into memory!\")\n \n fsUTM = field.to_crs(nir.spatial_ref.crs_wkt)\n\n # Crop to our ROI and apply scaling and masking\n nir_cropped = nir.rio.clip(fsUTM.geometry.values, fsUTM.crs, all_touched=True)\n red_cropped = red.rio.clip(fsUTM.geometry.values, fsUTM.crs, all_touched=True)\n blue_cropped = blue.rio.clip(fsUTM.geometry.values, fsUTM.crs, all_touched=True)\n fmask_cropped = fmask.rio.clip(fsUTM.geometry.values, fsUTM.crs, all_touched=True)\n \n #print('Cropped') \n \n # Fix Scaling\n nir_cropped_scaled = scaling(nir_cropped)\n red_cropped_scaled = scaling(red_cropped)\n blue_cropped_scaled = scaling(blue_cropped)\n\n\n # Generate EVI\n \n evi_cropped = calc_evi(red_cropped_scaled, blue_cropped_scaled, nir_cropped_scaled)\n\n #print('EVI Calculated')\n \n # Remove any observations that are entirely fill value\n if np.nansum(evi_cropped.data) == 0.0:\n print(f\"File: {h[0].split('/')[-1].rsplit('.', 1)[0]} was entirely fill values and will not be exported.\")\n continue\n \n evi_cropped.rio.to_raster(raster_path=f'{out_folder}{outName}', driver='COG')\n \n print(f\"Processing file {j+1} of {len(hls_results_urls)}\")\n \n\nProcessing file 1 of 78\nProcessing file 2 of 78\nProcessing file 3 of 78\nProcessing file 4 of 78\nProcessing file 5 of 78\nProcessing file 6 of 78\nProcessing file 7 of 78\nProcessing file 8 of 78\nProcessing file 9 of 78\nProcessing file 10 of 78\nHLS.S30.T10TEK.2021142T185921.v2.0_EVI_cropped.tif has already been processed and is available in this directory, moving to next file.\nProcessing file 12 of 78\nProcessing file 13 of 78\nProcessing file 14 of 78\nProcessing file 15 of 78\nProcessing file 16 of 78\nProcessing file 17 of 78\nProcessing file 18 of 78\nProcessing file 19 of 78\nProcessing file 20 of 78\nProcessing file 21 of 78\nProcessing file 22 of 78\nProcessing file 23 of 78\nProcessing file 24 of 78\nProcessing file 25 of 78\nProcessing file 26 of 78\nProcessing file 27 of 78\nProcessing file 28 of 78\nProcessing file 29 of 78\nProcessing file 30 of 78\nProcessing file 31 of 78\nProcessing file 32 of 78\nProcessing file 33 of 78\nProcessing file 34 of 78\nProcessing file 35 of 78\nProcessing file 36 of 78\nProcessing file 37 of 78\nProcessing file 38 of 78\nProcessing file 39 of 78\nProcessing file 40 of 78\nProcessing file 41 of 78\nProcessing file 42 of 78\nProcessing file 43 of 78\nProcessing file 44 of 78\nProcessing file 45 of 78\nProcessing file 46 of 78\nProcessing file 47 of 78\nProcessing file 48 of 78\nProcessing file 49 of 78\nProcessing file 50 of 78\nProcessing file 51 of 78\nProcessing file 52 of 78\nProcessing file 53 of 78\nProcessing file 54 of 78\nProcessing file 55 of 78\nProcessing file 56 of 78\nProcessing file 57 of 78\nProcessing file 58 of 78\nProcessing file 59 of 78\nProcessing file 60 of 78\nProcessing file 61 of 78\nProcessing file 62 of 78\nProcessing file 63 of 78\nProcessing file 64 of 78\nProcessing file 65 of 78\nProcessing file 66 of 78\nProcessing file 67 of 78\nProcessing file 68 of 78\nProcessing file 69 of 78\nProcessing file 70 of 78\nProcessing file 71 of 78\nProcessing file 72 of 78\nProcessing file 73 of 78\nProcessing file 74 of 78\nProcessing file 75 of 78\nProcessing file 76 of 78\nProcessing file 77 of 78\nProcessing file 78 of 78\n\n\nNow there should be multiple COGs exported to your working directory, that will be used in Section 6 to stack into a time series." }, { - "objectID": "appendix/authentication.html#earthdata-login", - "href": "appendix/authentication.html#earthdata-login", - "title": "Authentication", - "section": "", - "text": "An Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\nWe recommend authentication using earthaccess Python package, which you can also call from R. Please check the how do I find data using code on how to utilize the earthaccess package.\nAuthentication is also possible without the earthaccess package. You will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. The following code creates a hidden .netrc file (_netrc for Windows OS) with Earthdata login credentials in your home directory. This file is needed to access NASA Earthdata assets from scripting environments like Python and R. Below is code describing this process.\n\nPythonR\n\n\nPlease see this 2021 Cloud Hackathon tutorial for Python code.\n\n\nFor authentication in R, please run the code below:\n\n\n\n[R code]\n\n# Required packages for this script\npackages = c('sys', 'getPass')\n\n# Identify missing (not installed) packages\nnew.packages = packages[!(packages %in% installed.packages()[,\"Package\"])]\n\n# Install missing packages\nif(length(new.packages)) install.packages(new.packages, repos='http://cran.rstudio.com/')\n\n# Load packages into R\nlibrary(sys)\nlibrary(getPass)\n\n# Determine OS and associated netrc file \nnetrc_type <- if(.Platform$OS.type == \"windows\") \"_netrc\" else \".netrc\" # Windows OS uses _netrc file\n\n# Specify path to user profile \nup <- file.path(Sys.getenv(\"USERPROFILE\")) # Retrieve user directory (for netrc file)\n\n# Below, HOME and Userprofile directories are set. \n\nif (up == \"\") {\n up <- Sys.getenv(\"HOME\") \n Sys.setenv(\"userprofile\" = up)\n if (up == \"\") {\n cat('USERPROFILE/HOME directories need to be set up. Please type sys.setenv(\"HOME\" = \"YOURDIRECTORY\") or sys.setenv(\"USERPROFILE\" = \"YOURDIRECTORY\") in your console and type your USERPROFILE/HOME direcory instead of \"YOURDIRECTORY\". Next, run the code chunk again.')\n }\n} else {Sys.setenv(\"HOME\" = up)} \n\nnetrc_path <- file.path(up, netrc_type, fsep = .Platform$file.sep) # Path to netrc file\n\n# Create a netrc file if one does not exist already\nif (file.exists(netrc_path) == FALSE || grepl(\"urs.earthdata.nasa.gov\", readLines(netrc_path)) == FALSE) {\n netrc_conn <- file(netrc_path)\n \n # User will be prompted for NASA Earthdata Login Username and Password below\n writeLines(c(\"machine urs.earthdata.nasa.gov\",\n sprintf(\"login %s\", getPass(msg = \"Enter NASA Earthdata Login Username \\n (An account can be Created at urs.earthdata.nasa.gov):\")),\n sprintf(\"password %s\", getPass(msg = \"Enter NASA Earthdata Login Password:\"))), netrc_conn)\n close(netrc_conn)\n}else{\n i <- 0 \n for (f in readLines(netrc_path)){\n i <- i + 1\n if (f ==\"machine urs.earthdata.nasa.gov\"){\n username <- strsplit(readLines(netrc_path)[i+1], \" \")[[1]][2]\n un <- getPass(msg = paste0(\"Is your NASA Earthdata Login Username: \", username, \"\\n\\n Type yes or no.\"))\n if (tolower(un) == 'yes'){\n tx <- gsub(readLines(netrc_path)[i+2], sprintf(\"password %s\", getPass(msg = \"Enter NASA Earthdata Login Password:\")), readLines(netrc_path))\n writeLines(tx, netrc_path)\n rm(username, un, tx, f, i)\n }else{\n user <- gsub(readLines(netrc_path)[i+1], sprintf(\"login %s\", getPass(msg = \"Enter NASA Earthdata Login Username:\")), readLines(netrc_path))\n tx <- gsub(readLines(netrc_path)[i+2], sprintf(\"password %s\", getPass(msg = \"Enter NASA Earthdata Login Password:\")), readLines(netrc_path))\n writeLines(tx, netrc_path)\n rm(username, un, user, tx, f, i)\n \n }\n break\n }\n }\n}" + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#stacking-hls-data", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#stacking-hls-data", + "title": "Observing Seasonality in Agricultural Areas", + "section": "6. Stacking HLS Data ", + "text": "6. Stacking HLS Data \nIn this section we will open multiple HLS-derived EVI COGs and stack them into an xarray data array along the time dimension. First list the files we created in the /data/ directory.\n\n6.1 Open and Stack COGs\n\nevi_files = [os.path.abspath(o) for o in os.listdir(out_folder) if o.endswith('EVI_cropped.tif')] # List EVI COGs\n\nCreate a time index as an xarray variable from the filenames.\n\ndef time_index_from_filenames(evi_files):\n '''\n Helper function to create a pandas DatetimeIndex\n '''\n return [datetime.strptime(f.split('.')[-4], '%Y%jT%H%M%S') for f in evi_files]\n\ntime = xr.Variable('time', time_index_from_filenames(evi_files))\n\nNext, the cropped HLS COG files are being read using rioxarray and a time series stack is created using xarray.\n\n#evi_ts = evi_ts.squeeze('band', drop=True)\nchunks=dict(band=1, x=512, y=512)\n\nevi_ts = xr.concat([rxr.open_rasterio(f, mask_and_scale=True, chunks=chunks).squeeze('band', drop=True) for f in evi_files], dim=time)\nevi_ts.name = 'EVI'\n\n\nevi_ts = evi_ts.sortby(evi_ts.time)\nevi_ts\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray 'EVI' (time: 78, y: 59, x: 38)>\ndask.array<getitem, shape=(78, 59, 38), dtype=float32, chunksize=(1, 59, 38), chunktype=numpy.ndarray>\nCoordinates:\n * x (x) float64 5.802e+05 5.802e+05 ... 5.812e+05 5.813e+05\n * y (y) float64 4.419e+06 4.419e+06 ... 4.417e+06 4.417e+06\n spatial_ref int32 0\n * time (time) datetime64[ns] 2021-05-02T18:59:11 ... 2021-09-29T19:...\nAttributes: (12/33)\n ACCODE: Lasrc; Lasrc\n AREA_OR_POINT: Area\n arop_ave_xshift(meters): 0, 0\n arop_ave_yshift(meters): 0, 0\n arop_ncp: 0, 0\n arop_rmse(meters): 0, 0\n ... ...\n SPATIAL_RESOLUTION: 30\n TIRS_SSM_MODEL: FINAL; FINAL\n TIRS_SSM_POSITION_STATUS: ESTIMATED; ESTIMATED\n ULX: 499980\n ULY: 4500000\n USGS_SOFTWARE: LPGS_15.4.0xarray.DataArray'EVI'time: 78y: 59x: 38dask.array<chunksize=(1, 59, 38), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n683.11 kiB\n8.76 kiB\n\n\nShape\n(78, 59, 38)\n(1, 59, 38)\n\n\nDask graph\n78 chunks in 314 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\nCoordinates: (4)x(x)float645.802e+05 5.802e+05 ... 5.813e+05array([580155., 580185., 580215., 580245., 580275., 580305., 580335., 580365.,\n 580395., 580425., 580455., 580485., 580515., 580545., 580575., 580605.,\n 580635., 580665., 580695., 580725., 580755., 580785., 580815., 580845.,\n 580875., 580905., 580935., 580965., 580995., 581025., 581055., 581085.,\n 581115., 581145., 581175., 581205., 581235., 581265.])y(y)float644.419e+06 4.419e+06 ... 4.417e+06array([4418535., 4418505., 4418475., 4418445., 4418415., 4418385., 4418355.,\n 4418325., 4418295., 4418265., 4418235., 4418205., 4418175., 4418145.,\n 4418115., 4418085., 4418055., 4418025., 4417995., 4417965., 4417935.,\n 4417905., 4417875., 4417845., 4417815., 4417785., 4417755., 4417725.,\n 4417695., 4417665., 4417635., 4417605., 4417575., 4417545., 4417515.,\n 4417485., 4417455., 4417425., 4417395., 4417365., 4417335., 4417305.,\n 4417275., 4417245., 4417215., 4417185., 4417155., 4417125., 4417095.,\n 4417065., 4417035., 4417005., 4416975., 4416945., 4416915., 4416885.,\n 4416855., 4416825., 4416795.])spatial_ref()int320crs_wkt :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not specified (based on WGS 84 spheroid)projected_crs_name :UTM Zone 10, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-123.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :580140.0 30.0 0.0 4418550.0 0.0 -30.0array(0)time(time)datetime64[ns]2021-05-02T18:59:11 ... 2021-09-...array(['2021-05-02T18:59:11.000000000', '2021-05-04T18:49:19.000000000',\n '2021-05-07T18:59:19.000000000', '2021-05-08T18:44:47.000000000',\n '2021-05-09T18:49:21.000000000', '2021-05-12T18:59:21.000000000',\n '2021-05-14T18:49:19.000000000', '2021-05-15T18:51:03.000000000',\n '2021-05-17T18:59:19.000000000', '2021-05-19T18:49:21.000000000',\n '2021-05-22T18:59:21.000000000', '2021-05-24T18:44:57.000000000',\n '2021-05-24T18:49:19.000000000', '2021-05-27T18:59:19.000000000',\n '2021-05-29T18:49:21.000000000', '2021-05-31T18:51:12.000000000',\n '2021-06-01T18:59:21.000000000', '2021-06-03T18:49:19.000000000',\n '2021-06-06T18:59:19.000000000', '2021-06-08T18:49:21.000000000',\n '2021-06-09T18:45:05.000000000', '2021-06-11T18:59:21.000000000',\n '2021-06-13T18:49:19.000000000', '2021-06-16T18:51:18.000000000',\n '2021-06-16T18:59:19.000000000', '2021-06-18T18:49:21.000000000',\n '2021-06-21T18:59:21.000000000', '2021-06-23T18:49:19.000000000',\n '2021-06-25T18:45:09.000000000', '2021-06-26T18:59:19.000000000',\n '2021-06-28T18:49:21.000000000', '2021-07-01T18:59:21.000000000',\n '2021-07-02T18:51:21.000000000', '2021-07-06T18:59:19.000000000',\n '2021-07-08T18:49:21.000000000', '2021-07-11T18:45:11.000000000',\n '2021-07-11T18:59:21.000000000', '2021-07-13T18:49:19.000000000',\n '2021-07-16T18:59:19.000000000', '2021-07-18T18:49:21.000000000',\n '2021-07-18T18:51:24.000000000', '2021-07-21T18:59:21.000000000',\n '2021-07-23T18:49:19.000000000', '2021-07-26T18:59:19.000000000',\n '2021-07-28T18:49:21.000000000', '2021-07-31T18:59:21.000000000',\n '2021-08-02T18:49:19.000000000', '2021-08-03T18:51:32.000000000',\n '2021-08-05T18:59:19.000000000', '2021-08-07T18:49:21.000000000',\n '2021-08-10T18:59:21.000000000', '2021-08-12T18:45:24.000000000',\n '2021-08-12T18:49:19.000000000', '2021-08-15T18:59:19.000000000',\n '2021-08-17T18:49:21.000000000', '2021-08-19T18:51:37.000000000',\n '2021-08-20T18:59:21.000000000', '2021-08-22T18:49:19.000000000',\n '2021-08-25T18:59:19.000000000', '2021-08-27T18:49:21.000000000',\n '2021-08-28T18:45:29.000000000', '2021-08-30T18:59:21.000000000',\n '2021-09-01T18:49:19.000000000', '2021-09-04T18:51:42.000000000',\n '2021-09-04T18:59:09.000000000', '2021-09-06T18:49:21.000000000',\n '2021-09-09T18:59:31.000000000', '2021-09-11T18:49:29.000000000',\n '2021-09-13T18:45:33.000000000', '2021-09-14T18:59:29.000000000',\n '2021-09-16T18:50:21.000000000', '2021-09-19T19:00:51.000000000',\n '2021-09-20T18:51:45.000000000', '2021-09-21T18:49:59.000000000',\n '2021-09-24T19:00:29.000000000', '2021-09-26T18:51:31.000000000',\n '2021-09-29T18:45:37.000000000', '2021-09-29T19:02:01.000000000'],\n dtype='datetime64[ns]')Indexes: (3)xPandasIndexPandasIndex(Index([580155.0, 580185.0, 580215.0, 580245.0, 580275.0, 580305.0, 580335.0,\n 580365.0, 580395.0, 580425.0, 580455.0, 580485.0, 580515.0, 580545.0,\n 580575.0, 580605.0, 580635.0, 580665.0, 580695.0, 580725.0, 580755.0,\n 580785.0, 580815.0, 580845.0, 580875.0, 580905.0, 580935.0, 580965.0,\n 580995.0, 581025.0, 581055.0, 581085.0, 581115.0, 581145.0, 581175.0,\n 581205.0, 581235.0, 581265.0],\n dtype='float64', name='x'))yPandasIndexPandasIndex(Index([4418535.0, 4418505.0, 4418475.0, 4418445.0, 4418415.0, 4418385.0,\n 4418355.0, 4418325.0, 4418295.0, 4418265.0, 4418235.0, 4418205.0,\n 4418175.0, 4418145.0, 4418115.0, 4418085.0, 4418055.0, 4418025.0,\n 4417995.0, 4417965.0, 4417935.0, 4417905.0, 4417875.0, 4417845.0,\n 4417815.0, 4417785.0, 4417755.0, 4417725.0, 4417695.0, 4417665.0,\n 4417635.0, 4417605.0, 4417575.0, 4417545.0, 4417515.0, 4417485.0,\n 4417455.0, 4417425.0, 4417395.0, 4417365.0, 4417335.0, 4417305.0,\n 4417275.0, 4417245.0, 4417215.0, 4417185.0, 4417155.0, 4417125.0,\n 4417095.0, 4417065.0, 4417035.0, 4417005.0, 4416975.0, 4416945.0,\n 4416915.0, 4416885.0, 4416855.0, 4416825.0, 4416795.0],\n dtype='float64', name='y'))timePandasIndexPandasIndex(DatetimeIndex(['2021-05-02 18:59:11', '2021-05-04 18:49:19',\n '2021-05-07 18:59:19', '2021-05-08 18:44:47',\n '2021-05-09 18:49:21', '2021-05-12 18:59:21',\n '2021-05-14 18:49:19', '2021-05-15 18:51:03',\n '2021-05-17 18:59:19', '2021-05-19 18:49:21',\n '2021-05-22 18:59:21', '2021-05-24 18:44:57',\n '2021-05-24 18:49:19', '2021-05-27 18:59:19',\n '2021-05-29 18:49:21', '2021-05-31 18:51:12',\n '2021-06-01 18:59:21', '2021-06-03 18:49:19',\n '2021-06-06 18:59:19', '2021-06-08 18:49:21',\n '2021-06-09 18:45:05', '2021-06-11 18:59:21',\n '2021-06-13 18:49:19', '2021-06-16 18:51:18',\n '2021-06-16 18:59:19', '2021-06-18 18:49:21',\n '2021-06-21 18:59:21', '2021-06-23 18:49:19',\n '2021-06-25 18:45:09', '2021-06-26 18:59:19',\n '2021-06-28 18:49:21', '2021-07-01 18:59:21',\n '2021-07-02 18:51:21', '2021-07-06 18:59:19',\n '2021-07-08 18:49:21', '2021-07-11 18:45:11',\n '2021-07-11 18:59:21', '2021-07-13 18:49:19',\n '2021-07-16 18:59:19', '2021-07-18 18:49:21',\n '2021-07-18 18:51:24', '2021-07-21 18:59:21',\n '2021-07-23 18:49:19', '2021-07-26 18:59:19',\n '2021-07-28 18:49:21', '2021-07-31 18:59:21',\n '2021-08-02 18:49:19', '2021-08-03 18:51:32',\n '2021-08-05 18:59:19', '2021-08-07 18:49:21',\n '2021-08-10 18:59:21', '2021-08-12 18:45:24',\n '2021-08-12 18:49:19', '2021-08-15 18:59:19',\n '2021-08-17 18:49:21', '2021-08-19 18:51:37',\n '2021-08-20 18:59:21', '2021-08-22 18:49:19',\n '2021-08-25 18:59:19', '2021-08-27 18:49:21',\n '2021-08-28 18:45:29', '2021-08-30 18:59:21',\n '2021-09-01 18:49:19', '2021-09-04 18:51:42',\n '2021-09-04 18:59:09', '2021-09-06 18:49:21',\n '2021-09-09 18:59:31', '2021-09-11 18:49:29',\n '2021-09-13 18:45:33', '2021-09-14 18:59:29',\n '2021-09-16 18:50:21', '2021-09-19 19:00:51',\n '2021-09-20 18:51:45', '2021-09-21 18:49:59',\n '2021-09-24 19:00:29', '2021-09-26 18:51:31',\n '2021-09-29 18:45:37', '2021-09-29 19:02:01'],\n dtype='datetime64[ns]', name='time', freq=None))Attributes: (33)ACCODE :Lasrc; LasrcAREA_OR_POINT :Areaarop_ave_xshift(meters) :0, 0arop_ave_yshift(meters) :0, 0arop_ncp :0, 0arop_rmse(meters) :0, 0arop_s2_refimg :NONEcloud_coverage :0HLS_PROCESSING_TIME :2021-10-14T22:05:21ZHORIZONTAL_CS_NAME :UTM, WGS84, UTM ZONE 10; UTM, WGS84, UTM ZONE 10L1_PROCESSING_TIME :2021-05-18T02:30:51Z; 2021-05-18T02:30:41ZLANDSAT_PRODUCT_ID :LC08_L1TP_044032_20210508_20210518_02_T1; LC08_L1TP_044033_20210508_20210518_02_T1LANDSAT_SCENE_ID :LC80440322021128LGN00; LC80440332021128LGN00long_name :EVIMEAN_SUN_AZIMUTH_ANGLE :137.102321035166MEAN_SUN_ZENITH_ANGLE :28.615823781336MEAN_VIEW_AZIMUTH_ANGLE :105.923057001037MEAN_VIEW_ZENITH_ANGLE :5.72350593998171NBAR_SOLAR_ZENITH :27.1549084508815NCOLS :3660NROWS :3660OVR_RESAMPLING_ALG :NEARESTPROCESSING_LEVEL :L1TP; L1TPSENSING_TIME :2021-05-08T18:44:47.4233880Z; 2021-05-08T18:45:11.3101920ZSENSOR :OLI_TIRS; OLI_TIRSSENTINEL2_TILEID :10TEKspatial_coverage :55SPATIAL_RESOLUTION :30TIRS_SSM_MODEL :FINAL; FINALTIRS_SSM_POSITION_STATUS :ESTIMATED; ESTIMATEDULX :499980ULY :4500000USGS_SOFTWARE :LPGS_15.4.0\n\n\n\n\n6.2 Visualize Stacked Time Series\nBelow, use the hvPlot and holoviews packages to create an interactive time series plot of the HLS derived EVI data.Basemap layer is also added to provide better context of the areas surrounding our region of interest.\n\n# This cell generates a warning\nimport warnings\nwarnings.filterwarnings('ignore')\n\n\n# set the x, y, and z (groupby) dimensions, add a colormap/bar and other parameters.\ntitle = 'HLS-derived EVI over agricultural fields in northern California'\nevi_ts.hvplot.image(x='x', y='y', groupby = 'time', frame_height=500, frame_width= 500, cmap='YlGn', geo=True, tiles = 'EsriImagery',)\n\n\n\n\n\n \n\n\n\n\nLooking at the time series above, the farm fields are pretty stable in terms of EVI during our temporal range. The slow decrease in EVI as we move toward Fall season could show these fields are having some sort of trees rather than crops. I encourage you to expand your temporal range to learn more about the EVI annual and seasonal changes.\nSince the data is in an xarray we can intuitively slice or reduce the dataset. Let’s select a single time slice from the EVI variable.\nYou can use slicing to plot data only for a specific observation, for example.\n\ntitle = 'HLS-derived EVI over agricultural fields in northern California'\n# evi_cropped.hvplot.image(aspect='equal', cmap='YlGn', frame_width=300).opts(title=f'HLS-derived EVI, {evi_cropped.SENSING_TIME}', clabel='EVI')\n\nevi_ts.isel(time=1).hvplot.image(x='x', y='y', cmap='YlGn', geo=True, tiles = 'EsriImagery', frame_height=500, frame_width= 500).opts(title=f'{title}, {evi_ts.isel(time=4).SENSING_TIME}')\n\n\n\n\n\n \n\n\n\n\nNow, plot the time series as boxplots showing the distribution of EVI values for our farm field.\n\nevi_ts.hvplot.box('EVI', by=['time'], rot=90, box_fill_color='lightblue', width=800, height=600).opts(ylim=(0,1.0))\n\n\n\n\n\n \n\n\n\n\nThe statistics shows a relatively stable green status in these fields during mid May to the end of September 2021." }, { - "objectID": "appendix/authentication.html#aws-credentials", - "href": "appendix/authentication.html#aws-credentials", - "title": "Authentication", - "section": "AWS Credentials", - "text": "AWS Credentials\ncoming soon" + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#export-statistics", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#export-statistics", + "title": "Observing Seasonality in Agricultural Areas", + "section": "7. Export Statistics", + "text": "7. Export Statistics\nNext, calculate statistics for each observation and export to CSV.\n\n# xarray allows you to easily calculate a number of statistics\nevi_min = evi_ts.min(('y', 'x'))\nevi_max = evi_ts.max(('y', 'x'))\nevi_mean = evi_ts.mean(('y', 'x'))\nevi_sd = evi_ts.std(('y', 'x'))\nevi_count = evi_ts.count(('y', 'x'))\nevi_median = evi_ts.median(('y', 'x'))\n\nWe now have the mean and standard deviation for each time slice as well as the maximum and minimum values. Let’s do some plotting! We will use the hvPlot package to create simple but interactive charts/plots. Hover your curser over the visualization to see the data values.\n\nevi_mean.hvplot.line()\n\n\n\n\n\n \n\n\n\n\n\n# Ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Combine line plots for different statistics\nstats = (evi_mean.hvplot.line(height=350, width=450, line_width=1.5, color='red', grid=True, padding=0.05).opts(title='Mean')+ \n evi_sd.hvplot.line(height=350, width=450, line_width=1.5, color='red', grid=True, padding=0.05).opts(title='Standard Deviation')\n + evi_max.hvplot.line(height=350, width=450, line_width=1.5, color='red', grid=True, padding=0.05).opts(title='Max') + \n evi_min.hvplot.line(height=350, width=450, line_width=1.5, color='red', grid=True, padding=0.05).opts(title='Min')).cols(2)\nstats\n\n\n\n\n\n \n\n\n\n\nRemember that these graphs are also interactive–hover over the line to see the value for a given date.\nFinally, create a pandas dataframe with the statistics, and export to a CSV file.\n\n# Create pandas dataframe from dictionary\ndf = pd.DataFrame({'Min EVI': evi_min, 'Max EVI': evi_max, \n 'Mean EVI': evi_mean, 'Standard Deviation EVI': evi_sd, \n 'Median EVI': evi_median, 'Count': evi_count})\n\n\ndf.index = evi_ts.time.data # Set the observation date as the index\ndf.to_csv(f'{out_folder}HLS-Derived_EVI_Stats.csv', index=True) # Unhash to export to CSV\n\n\nNow remove the output files to clean up the workspace.\n\nimport shutil\nshutil.rmtree(os.path.abspath(out_folder)) # Execute once ready to remove output files\n\nSuccess! You have now not only learned how to get started with HLS V2.0 data, but have also learned how to navigate cloud-native data using STAC, how to access subsets of COGs, and how to write COGs for your own outputs. Using this jupyter notebook as a workflow, you should now be able to switch to your specific region of interest and re-run the notebook. Good Luck!" }, { - "objectID": "glossary.html", - "href": "glossary.html", - "title": "Glossary, Cheatsheets, & Guides", - "section": "", - "text": "A new paradigm brings new terminology you will encounter while you learn about cloud data access and computing.\nThis handy cheatsheet, Cloud Terminology 101, defines commonly used cloud computing terms and phrases.\n\n\nPublished Google Slide\nMore specific to working with NASA Earthdata in the cloud, this Workflow Terminology Cheatsheet defines terms associated with NASA Earthdata, cloud-optimized data, and some open software science tools you may encounter. A full list of NASA Eathdata-specific terms are found in the NASA Earthdata Glossary.\n\n\nPublished Google Slide" + "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#contact-info", + "href": "tutorials/Observing_Seasonal_Ag_Changes.html#contact-info", + "title": "Observing Seasonality in Agricultural Areas", + "section": "Contact Info", + "text": "Contact Info\nEmail: LPDAAC@usgs.gov\nVoice: +1-866-573-3222\nOrganization: Land Processes Distributed Active Archive Center (LP DAAC)¹\nWebsite: https://lpdaac.usgs.gov/\nDate last modified: 09-08-2023\n¹Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I." }, { - "objectID": "glossary.html#glossary", - "href": "glossary.html#glossary", - "title": "Glossary, Cheatsheets, & Guides", + "objectID": "tutorials/matlab.html", + "href": "tutorials/matlab.html", + "title": "MATLAB Access Single NASA EarthData L2 NetCDF", "section": "", - "text": "A new paradigm brings new terminology you will encounter while you learn about cloud data access and computing.\nThis handy cheatsheet, Cloud Terminology 101, defines commonly used cloud computing terms and phrases.\n\n\nPublished Google Slide\nMore specific to working with NASA Earthdata in the cloud, this Workflow Terminology Cheatsheet defines terms associated with NASA Earthdata, cloud-optimized data, and some open software science tools you may encounter. A full list of NASA Eathdata-specific terms are found in the NASA Earthdata Glossary.\n\n\nPublished Google Slide" - }, - { - "objectID": "glossary.html#cheatsheets-guides", - "href": "glossary.html#cheatsheets-guides", - "title": "Glossary, Cheatsheets, & Guides", - "section": "Cheatsheets & Guides", - "text": "Cheatsheets & Guides\nHow are all of these terms and concepts related? And how can you begin to put them to work?\nHere are some cheatsheets and guides to help visualize what working with NASA Earthdata in the cloud can look like, and how to get started.\nAll slides and cheatsheets are available for re-use and remix! Let us know what you think! We welcome your input so we can continue to improve and update these guides. Slides are credited for each deck; Cheatsheets development have been led by Catalina Oaida Taglialatela and Cassie Nickles (PO.DAAC) in Spring 2022.\n(Internal link).\n\nWhat is the NASA Earthdata Cloud?\nNASA Earthdata Cloud is the NASA archive of Earth observations and is hosted in Amazon Web Services (AWS) cloud with DAAC tools and services built for use “next to the data.” The NASA DAACs (data centers) are currently transitioning to this cloud-based environment. The cloud offers a scalable and effective way to address storage, network, and data movement concerns while offering a tremendous amount of flexibility to the user. Particularly if working with large data volumes, data access and processing would be more efficient if workflows are taking place in the cloud, avoiding having to download large data volumes. Data download will continue to be freely available to users, from the Earthdata Cloud archive.\n\n\nPublished Google Slide\n\n\nCloud Access Pathways\nThree pathway examples, to interact and access data (and services) from and within the NASA Earthdata Cloud, are illustrated in the diagram. Green arrows and icons indicate working locally, after downloading data to your local machine, servers, or compute/storage space. Orange arrows and icons highlight a workflow within the cloud, setting up your own AWS EC2 cloud instance, or virtual machine, in the cloud next to the data. Blue arrows and icons also indicate a within the cloud workflow, through shareable cloud environments such as Binder or JupyterHub set up in an AWS cloud region. Note that each of these may have a range of cost models. EOSDIS data are being stored in the us-west-2 region of AWS cloud; we recommend setting up your cloud computing environment in the same region as the data for free and easy in-cloud access.\n\n\nPublished Google Slide\nA note on costing: What is free and what do I have to budget for, now that data is archived in the cloud?\n\nDownloading data from the Earthdata Cloud archive in AWS, to your local computer environment or local storage (e.g. servers) is and will continue to be free for the user.\nAccessing the data directly in the cloud (from us-west-2 S3 region) is free. Users will need a NASA Earthdata Login account and AWS credentials to access, but there is no cost associated with these authentication steps, which are in place for security reasons.\nAccessing data in the cloud via EOSDIS or DAAC cloud-based tools and services such as the CMR API, Harmony API, OPenDAP API (from us-west-2 S3 region) is free to the user. Having the tools and services “next to the data” in the cloud enables DAACs to support data reduction and transformation, more efficiently, on behalf of the user, so users only access the data they need.\nCloud computing environments (i.e. virtual machines in the cloud) for working with data in the cloud (beyond direct or via services provided access) such as data analysis or running models with the data, is user responsibility, and should be considered in budgeting. I.e. User would need to set up a cloud compute environment (such as an EC2 instance or JupyterLab) and are responsible for any storage and computing costs.\n\nThis means that even though direct data access in the cloud is free to the user, they would first need to have a cloud computing environment/machine to execute the data access step from, and then continue their analysis.\nDepending on whether that cloud environment is provided by the user themselves, user’s institution, community hubs like Pangeo or NASA Openscapes JupyterLab sandbox, this element of the workflow may require user accountability, budgeting and user financial maintenance.\n\n\n\n\nGetting Started Roadmap\n\nCloud Workflow\nThe following is a conceptual roadmap for users getting started with NASA Earth Observations cloud-archived data using an in-cloud workflow (i.e. bringing user code into the cloud, avoiding data download and performing data workflows “next to the data”).\n\n\nPublished Google Slide\n\n\nLocal Workflow\nThe following is a conceptual roadmap for users getting started with NASA Earth Observations cloud-archived data using a local machine (e.g. laptop) workflow, as data storage and computational work.\n\n\nPublished Google Slide\n\n\n\nTools & Services Roadmap\nBelow is a practical guide for learning about and selecting helpful tools or services for a given use case, focusing on how to find and access NASA Earthdata Cloud-archived data from local compute environment (e.g. laptop) or from a cloud computing workspace, with accompanying example tutorials. Once you follow your desired pathway, click on the respective blue notebook icon to get to the example tutorial. Note: these pathways are not exhaustive, there are many ways to accomplish these common steps, but these are some of our recommendations.\n\n\nPublished Google Slide\n\n\nWorkflow Cheatsheet\nThe following is a practical reference guide with links to tutorials and informational websites for users who are starting to take the conceptual pieces and explore and implement in their own workflows.\n\n\nPublished Google Slide" - }, - { - "objectID": "glossary.html#slides", - "href": "glossary.html#slides", - "title": "Glossary, Cheatsheets, & Guides", - "section": "Slides", - "text": "Slides\nSelected presentations about working with NASA Earthdata on the Cloud; for all presentations see nasa-openscapes.github.io > slides.\n\nNASA Earthdata Cloud: Myths, Truths, Questions\nby Amy Steiker, Kate Heightley (NSIDC) September 7, 2022\n\n\nNSIDC DAAC User Working Group\nby Andrew Barrett, Amy Steiker, Walt Meier, Jennie Roebuck, Mikala Beig, Luis Lopez, (NSIDC) May 20, 2022.\n\n\nNASA Earthdata Cloud & The Cloud Paradigm\nby Aaron Friesz (LP DAAC), April 2022." + "text": "This tutorial was written by Erin Robinson, Luis Lopez (NASA Openscapes) and Cedric Wannaz, Lisa Kempler (Mathworks). Also available as a 9-minute Video walk-through." }, { - "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html", - "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html", - "title": "PO.DAAC ECCO SSH", - "section": "", - "text": "Many of NASA’s current and legacy data collections are archive in netCDF4 format. By itself, netCDF4 are not cloud optimized and reading these files can take as long from a personal/local work environment as it takes to read the data from a working environment deployed in the cloud. Using Kerchunk, we can treat these files as cloud optimized assets by creating metadata json file describing existing netCDF4 files, their chunks, and where to access them. The json reference files can be read in using Zarr and Xarray for efficient reads and fast processing." + "objectID": "tutorials/matlab.html#launch-matlab-in-2i2c-jupyterhub", + "href": "tutorials/matlab.html#launch-matlab-in-2i2c-jupyterhub", + "title": "MATLAB Access Single NASA EarthData L2 NetCDF", + "section": "Launch MATLAB in 2i2c JupyterHub", + "text": "Launch MATLAB in 2i2c JupyterHub\nYou’re able to use MATLAB in the 2i2c JupyterHub; you are able to bring your own license. If you’d like to access MATLAB from the Amazon Web Services Console, see details at the bottom of this page.\n\n\n\nSelect MATLAB from the Server Options and press Start\n\n\n\n\n\nLaunch MATLAB from the JupyterHub interface by double-clicking\n\n\n\n\n\nEnter your MATLAB login information. NOTE: You will also need to be able to access your MATLAB account via MATLAB Online." }, { - "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#requirements", - "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#requirements", - "title": "PO.DAAC ECCO SSH", - "section": "Requirements", - "text": "Requirements\n\n1. AWS instance running in us-west-2\nNASA Earthdata Cloud data in S3 can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region.\n\n\n2. Earthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\n3. netrc File\nYou will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. For additional information see: Authentication for NASA Earthdata." + "objectID": "tutorials/matlab.html#credentials", + "href": "tutorials/matlab.html#credentials", + "title": "MATLAB Access Single NASA EarthData L2 NetCDF", + "section": "Credentials", + "text": "Credentials\nDirect S3 access is achieved by passing NASA supplied temporary credentials to AWS so we can interact with S3 objects from applicable EarthData Cloud buckets. For now, each NASA DAAC has different AWS credentials endpoints. Below are some of the credential endpoints to various DAACs:\n\nPODAAC: https://archive.podaac.earthdata.nasa.gov/s3credentials\nGESDISC: https://data.gesdisc.earthdata.nasa.gov/s3credentials\nLPDAAC: https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials\nORNLDAAC: https://data.ornldaac.earthdata.nasa.gov/s3credentials\nGHRC DAAC: https://data.ghrc.earthdata.nasa.gov/s3credentials\nNSIDC: https://data.nsidc.earthdatacloud.nasa.gov/s3credentials\n\nYou will need your EarthData login to access these links. Create an account here: https://urs.earthdata.nasa.gov/users/new." }, { - "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#import-required-packages", - "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#import-required-packages", - "title": "PO.DAAC ECCO SSH", - "section": "Import required packages", - "text": "Import required packages\n\nimport requests\nimport xarray as xr\nimport ujson\nimport s3fs\nimport fsspec\nfrom tqdm import tqdm\nfrom glob import glob\nimport os\nimport pathlib\nimport hvplot.xarray\n\nfrom kerchunk.hdf import SingleHdf5ToZarr\nfrom kerchunk.combine import MultiZarrToZarr\n\n# The xarray produced from the reference file throws a SerializationWarning for each variable. Will need to explore why\nimport warnings\nwarnings.simplefilter(\"ignore\")" - }, - { - "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#create-dask-client-to-process-the-output-json-file-in-parallel", - "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#create-dask-client-to-process-the-output-json-file-in-parallel", - "title": "PO.DAAC ECCO SSH", - "section": "Create Dask client to process the output json file in parallel", - "text": "Create Dask client to process the output json file in parallel\nGenerating the Kerchunk reference file can take some time depending on the internal structure of the data. Dask allows us to execute the reference file generation process in parallel, thus speeding up the overall process.\n\nimport dask\nfrom dask.distributed import Client\nclient = Client(n_workers=4)\nclient\n\n2022-05-11 15:27:29,674 - distributed.diskutils - INFO - Found stale lock file and directory '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/dask-worker-space/worker-mezhdsy7', purging\n/srv/conda/envs/notebook/lib/python3.9/contextlib.py:126: UserWarning: Creating scratch directories is taking a surprisingly long time. This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.\n next(self.gen)\n\n\n\n \n \n Client\n Client-ddf55e52-d13e-11ec-818c-b6609e8b92a4\n \n\n\n\nConnection method: Cluster object\nCluster type: distributed.LocalCluster\n\n\nDashboard: http://127.0.0.1:41805/status\n\n\n\n\n\nCluster Info\n\n\n\n\n\n\nLocalCluster\na24e60d3\n\n\n\nDashboard: http://127.0.0.1:41805/status\nWorkers: 4\n\n\nTotal threads: 4\nTotal memory: 15.18 GiB\n\n\nStatus: running\nUsing processes: True\n\n\n\n\n\nScheduler Info\n\n\n\n\n\n\nScheduler\nScheduler-8e045442-a409-4c3b-8c8c-a95470883931\n\n\n\nComm: tcp://127.0.0.1:36901\nWorkers: 4\n\n\nDashboard: http://127.0.0.1:41805/status\nTotal threads: 4\n\n\nStarted: Just now\nTotal memory: 15.18 GiB\n\n\n\n\n\n\nWorkers\n\n\n\n\n\nWorker: 0\n\n\n\nComm: tcp://127.0.0.1:34235\nTotal threads: 1\n\n\nDashboard: http://127.0.0.1:42845/status\nMemory: 3.80 GiB\n\n\nNanny: tcp://127.0.0.1:37927\n\n\n\nLocal directory: /home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/dask-worker-space/worker-869qv5xb\n\n\n\n\n\n\n\n\n\n\nWorker: 1\n\n\n\nComm: tcp://127.0.0.1:40997\nTotal threads: 1\n\n\nDashboard: http://127.0.0.1:41189/status\nMemory: 3.80 GiB\n\n\nNanny: tcp://127.0.0.1:35257\n\n\n\nLocal directory: /home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/dask-worker-space/worker-3mo0d80c\n\n\n\n\n\n\n\n\n\n\nWorker: 2\n\n\n\nComm: tcp://127.0.0.1:46429\nTotal threads: 1\n\n\nDashboard: http://127.0.0.1:42211/status\nMemory: 3.80 GiB\n\n\nNanny: tcp://127.0.0.1:34287\n\n\n\nLocal directory: /home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/dask-worker-space/worker-o2fvmao4\n\n\n\n\n\n\n\n\n\n\nWorker: 3\n\n\n\nComm: tcp://127.0.0.1:41615\nTotal threads: 1\n\n\nDashboard: http://127.0.0.1:41507/status\nMemory: 3.80 GiB\n\n\nNanny: tcp://127.0.0.1:43053\n\n\n\nLocal directory: /home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/dask-worker-space/worker-9u77hywd" - }, - { - "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#get-temporary-s3-credentials", - "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#get-temporary-s3-credentials", - "title": "PO.DAAC ECCO SSH", - "section": "Get temporary S3 credentials", - "text": "Get temporary S3 credentials\nTemporary S3 credentials need to be passed to AWS. Note, these credentials must be refreshed after 1 hour.\n\ns3_cred_endpoint = {\n 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n 'ornldaac':'https://data.ornldaac.earthdata.nasa.gov/s3credentials',\n 'gesdisc':'https://data.gesdisc.earthdata.nasa.gov/s3credentials'\n}\n\n\ndef get_temp_creds():\n temp_creds_url = s3_cred_endpoint['podaac']\n return requests.get(temp_creds_url).json()\n\n\ntemp_creds_req = get_temp_creds()" - }, - { - "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#direct-access-a-single-netcdf4-file", - "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#direct-access-a-single-netcdf4-file", - "title": "PO.DAAC ECCO SSH", - "section": "Direct Access a single netCDF4 file", - "text": "Direct Access a single netCDF4 file\nPass temporary credentials to our filesystem object to access the S3 assets\n\nfs = s3fs.S3FileSystem(\n anon=False,\n key=temp_creds_req['accessKeyId'],\n secret=temp_creds_req['secretAccessKey'],\n token=temp_creds_req['sessionToken']\n)\n\n\nurl = 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-01_ECCO_V4r4_latlon_0p50deg.nc'\n\n\ns3_file_obj = fs.open(url, mode='rb')\n\nTime how long it takes to directly access a cloud asset for comparisons later.\n\n%%time\n\nxr_ds = xr.open_dataset(s3_file_obj, chunks='auto', engine='h5netcdf')\nxr_ds\n\nCPU times: user 228 ms, sys: 8.51 ms, total: 237 ms\nWall time: 272 ms\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (time: 1, latitude: 360, longitude: 720, nv: 2)\nCoordinates:\n * time (time) datetime64[ns] 2015-01-16T12:00:00\n * latitude (latitude) float32 -89.75 -89.25 -88.75 ... 89.25 89.75\n * longitude (longitude) float32 -179.8 -179.2 -178.8 ... 179.2 179.8\n time_bnds (time, nv) datetime64[ns] dask.array<chunksize=(1, 2), meta=np.ndarray>\n latitude_bnds (latitude, nv) float32 dask.array<chunksize=(360, 2), meta=np.ndarray>\n longitude_bnds (longitude, nv) float32 dask.array<chunksize=(720, 2), meta=np.ndarray>\nDimensions without coordinates: nv\nData variables:\n SSH (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n SSHIBC (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n SSHNOIBC (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\nAttributes: (12/57)\n acknowledgement: This research was carried out by the Jet Pr...\n author: Ian Fenty and Ou Wang\n cdm_data_type: Grid\n comment: Fields provided on a regular lat-lon grid. ...\n Conventions: CF-1.8, ACDD-1.3\n coordinates_comment: Note: the global 'coordinates' attribute de...\n ... ...\n time_coverage_duration: P1M\n time_coverage_end: 2015-02-01T00:00:00\n time_coverage_resolution: P1M\n time_coverage_start: 2015-01-01T00:00:00\n title: ECCO Sea Surface Height - Monthly Mean 0.5 ...\n uuid: 088d03b8-4158-11eb-876b-0cc47a3f47f1xarray.DatasetDimensions:time: 1latitude: 360longitude: 720nv: 2Coordinates: (6)time(time)datetime64[ns]2015-01-16T12:00:00axis :Tbounds :time_bndscoverage_content_type :coordinatelong_name :center time of averaging periodstandard_name :timearray(['2015-01-16T12:00:00.000000000'], dtype='datetime64[ns]')latitude(latitude)float32-89.75 -89.25 ... 89.25 89.75axis :Ybounds :latitude_bndscomment :uniform grid spacing from -89.75 to 89.75 by 0.5coverage_content_type :coordinatelong_name :latitude at grid cell centerstandard_name :latitudeunits :degrees_northarray([-89.75, -89.25, -88.75, ..., 88.75, 89.25, 89.75], dtype=float32)longitude(longitude)float32-179.8 -179.2 ... 179.2 179.8axis :Xbounds :longitude_bndscomment :uniform grid spacing from -179.75 to 179.75 by 0.5coverage_content_type :coordinatelong_name :longitude at grid cell centerstandard_name :longitudeunits :degrees_eastarray([-179.75, -179.25, -178.75, ..., 178.75, 179.25, 179.75],\n dtype=float32)time_bnds(time, nv)datetime64[ns]dask.array<chunksize=(1, 2), meta=np.ndarray>comment :Start and end times of averaging period.coverage_content_type :coordinatelong_name :time bounds of averaging period\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n16 B\n16 B\n\n\nShape\n(1, 2)\n(1, 2)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\ndatetime64[ns]\nnumpy.ndarray\n\n\n\n\n\n\n\n\nlatitude_bnds\n\n\n(latitude, nv)\n\n\nfloat32\n\n\ndask.array<chunksize=(360, 2), meta=np.ndarray>\n\n\n\n\ncoverage_content_type :\n\ncoordinate\n\nlong_name :\n\nlatitude bounds grid cells\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n2.81 kiB\n2.81 kiB\n\n\nShape\n(360, 2)\n(360, 2)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nlongitude_bnds\n\n\n(longitude, nv)\n\n\nfloat32\n\n\ndask.array<chunksize=(720, 2), meta=np.ndarray>\n\n\n\n\ncoverage_content_type :\n\ncoordinate\n\nlong_name :\n\nlongitude bounds grid cells\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n5.62 kiB\n5.62 kiB\n\n\nShape\n(720, 2)\n(720, 2)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nData variables: (3)\n\n\n\n\n\nSSH\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nDynamic sea surface height anomaly\n\nstandard_name :\n\nsea_surface_height_above_geoid\n\nunits :\n\nm\n\ncomment :\n\nDynamic sea surface height anomaly above the geoid, suitable for comparisons with altimetry sea surface height data products that apply the inverse barometer (IB) correction. Note: SSH is calculated by correcting model sea level anomaly ETAN for three effects: a) global mean steric sea level changes related to density changes in the Boussinesq volume-conserving model (Greatbatch correction, see sterGloH), b) the inverted barometer (IB) effect (see SSHIBC) and c) sea level displacement due to sea-ice and snow pressure loading (see sIceLoad). SSH can be compared with the similarly-named SSH variable in previous ECCO products that did not include atmospheric pressure loading (e.g., Version 4 Release 3). Use SSHNOIBC for comparisons with altimetry data products that do NOT apply the IB correction.\n\nvalid_min :\n\n-1.8805772066116333\n\nvalid_max :\n\n1.4207719564437866\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.99 MiB\n0.99 MiB\n\n\nShape\n(1, 360, 720)\n(1, 360, 720)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSSHIBC\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nThe inverted barometer (IB) correction to sea surface height due to atmospheric pressure loading\n\nunits :\n\nm\n\ncomment :\n\nNot an SSH itself, but a correction to model sea level anomaly (ETAN) required to account for the static part of sea surface displacement by atmosphere pressure loading: SSH = SSHNOIBC - SSHIBC. Note: Use SSH for model-data comparisons with altimetry data products that DO apply the IB correction and SSHNOIBC for comparisons with altimetry data products that do NOT apply the IB correction.\n\nvalid_min :\n\n-0.30144819617271423\n\nvalid_max :\n\n0.5245633721351624\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.99 MiB\n0.99 MiB\n\n\nShape\n(1, 360, 720)\n(1, 360, 720)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSSHNOIBC\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nSea surface height anomaly without the inverted barometer (IB) correction\n\nunits :\n\nm\n\ncomment :\n\nSea surface height anomaly above the geoid without the inverse barometer (IB) correction, suitable for comparisons with altimetry sea surface height data products that do NOT apply the inverse barometer (IB) correction. Note: SSHNOIBC is calculated by correcting model sea level anomaly ETAN for two effects: a) global mean steric sea level changes related to density changes in the Boussinesq volume-conserving model (Greatbatch correction, see sterGloH), b) sea level displacement due to sea-ice and snow pressure loading (see sIceLoad). In ECCO Version 4 Release 4 the model is forced with atmospheric pressure loading. SSHNOIBC does not correct for the static part of the effect of atmosphere pressure loading on sea surface height (the so-called inverse barometer (IB) correction). Use SSH for comparisons with altimetry data products that DO apply the IB correction.\n\nvalid_min :\n\n-1.6654272079467773\n\nvalid_max :\n\n1.4550364017486572\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.99 MiB\n0.99 MiB\n\n\nShape\n(1, 360, 720)\n(1, 360, 720)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nAttributes: (57)acknowledgement :This research was carried out by the Jet Propulsion Laboratory, managed by the California Institute of Technology under a contract with the National Aeronautics and Space Administration.author :Ian Fenty and Ou Wangcdm_data_type :Gridcomment :Fields provided on a regular lat-lon grid. They have been mapped to the regular lat-lon grid from the original ECCO lat-lon-cap 90 (llc90) native model grid. SSH (dynamic sea surface height) = SSHNOIBC (dynamic sea surface without the inverse barometer correction) - SSHIBC (inverse barometer correction). The inverted barometer correction accounts for variations in sea surface height due to atmospheric pressure variations.Conventions :CF-1.8, ACDD-1.3coordinates_comment :Note: the global 'coordinates' attribute describes auxillary coordinates.creator_email :ecco-group@mit.educreator_institution :NASA Jet Propulsion Laboratory (JPL)creator_name :ECCO Consortiumcreator_type :groupcreator_url :https://ecco-group.orgdate_created :2020-12-18T09:39:51date_issued :2020-12-18T09:39:51date_metadata_modified :2021-03-15T22:07:49date_modified :2021-03-15T22:07:49geospatial_bounds_crs :EPSG:4326geospatial_lat_max :90.0geospatial_lat_min :-90.0geospatial_lat_resolution :0.5geospatial_lat_units :degrees_northgeospatial_lon_max :180.0geospatial_lon_min :-180.0geospatial_lon_resolution :0.5geospatial_lon_units :degrees_easthistory :Inaugural release of an ECCO Central Estimate solution to PO.DAACid :10.5067/ECG5M-SSH44institution :NASA Jet Propulsion Laboratory (JPL)instrument_vocabulary :GCMD instrument keywordskeywords :EARTH SCIENCE > OCEANS > SEA SURFACE TOPOGRAPHY > SEA SURFACE HEIGHT, EARTH SCIENCE SERVICES > MODELS > EARTH SCIENCE REANALYSES/ASSIMILATION MODELSkeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordslicense :Public Domainmetadata_link :https://cmr.earthdata.nasa.gov/search/collections.umm_json?ShortName=ECCO_L4_SSH_05DEG_MONTHLY_V4R4naming_authority :gov.nasa.jplplatform :ERS-1/2, TOPEX/Poseidon, Geosat Follow-On (GFO), ENVISAT, Jason-1, Jason-2, CryoSat-2, SARAL/AltiKa, Jason-3, AVHRR, Aquarius, SSM/I, SSMIS, GRACE, DTU17MDT, Argo, WOCE, GO-SHIP, MEOP, Ice Tethered Profilers (ITP)platform_vocabulary :GCMD platform keywordsprocessing_level :L4product_name :SEA_SURFACE_HEIGHT_mon_mean_2015-01_ECCO_V4r4_latlon_0p50deg.ncproduct_time_coverage_end :2018-01-01T00:00:00product_time_coverage_start :1992-01-01T12:00:00product_version :Version 4, Release 4program :NASA Physical Oceanography, Cryosphere, Modeling, Analysis, and Prediction (MAP)project :Estimating the Circulation and Climate of the Ocean (ECCO)publisher_email :podaac@podaac.jpl.nasa.govpublisher_institution :PO.DAACpublisher_name :Physical Oceanography Distributed Active Archive Center (PO.DAAC)publisher_type :institutionpublisher_url :https://podaac.jpl.nasa.govreferences :ECCO Consortium, Fukumori, I., Wang, O., Fenty, I., Forget, G., Heimbach, P., & Ponte, R. M. 2020. Synopsis of the ECCO Central Production Global Ocean and Sea-Ice State Estimate (Version 4 Release 4). doi:10.5281/zenodo.3765928source :The ECCO V4r4 state estimate was produced by fitting a free-running solution of the MITgcm (checkpoint 66g) to satellite and in situ observational data in a least squares sense using the adjoint methodstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventionsummary :This dataset provides monthly-averaged dynamic sea surface height interpolated to a regular 0.5-degree grid from the ECCO Version 4 Release 4 (V4r4) ocean and sea-ice state estimate. Estimating the Circulation and Climate of the Ocean (ECCO) state estimates are dynamically and kinematically-consistent reconstructions of the three-dimensional, time-evolving ocean, sea-ice, and surface atmospheric states. ECCO V4r4 is a free-running solution of a global, nominally 1-degree configuration of the MIT general circulation model (MITgcm) that has been fit to observations in a least-squares sense. Observational data constraints used in V4r4 include sea surface height (SSH) from satellite altimeters [ERS-1/2, TOPEX/Poseidon, GFO, ENVISAT, Jason-1,2,3, CryoSat-2, and SARAL/AltiKa]; sea surface temperature (SST) from satellite radiometers [AVHRR], sea surface salinity (SSS) from the Aquarius satellite radiometer/scatterometer, ocean bottom pressure (OBP) from the GRACE satellite gravimeter; sea-ice concentration from satellite radiometers [SSM/I and SSMIS], and in-situ ocean temperature and salinity measured with conductivity-temperature-depth (CTD) sensors and expendable bathythermographs (XBTs) from several programs [e.g., WOCE, GO-SHIP, Argo, and others] and platforms [e.g., research vessels, gliders, moorings, ice-tethered profilers, and instrumented pinnipeds]. V4r4 covers the period 1992-01-01T12:00:00 to 2018-01-01T00:00:00.time_coverage_duration :P1Mtime_coverage_end :2015-02-01T00:00:00time_coverage_resolution :P1Mtime_coverage_start :2015-01-01T00:00:00title :ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4)uuid :088d03b8-4158-11eb-876b-0cc47a3f47f1" - }, - { - "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#specify-a-list-of-s3-urls", - "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#specify-a-list-of-s3-urls", - "title": "PO.DAAC ECCO SSH", - "section": "Specify a list of S3 URLs", - "text": "Specify a list of S3 URLs\nData Collection: ECCO_L4_SSH_05DEG_MONTHLY_V4R4\nTime Range: 2015\n\nurls = ['s3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2014-12_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-01_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-02_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-03_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-04_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-05_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-06_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-07_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-08_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-09_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-10_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-11_ECCO_V4r4_latlon_0p50deg.nc',\n 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-12_ECCO_V4r4_latlon_0p50deg.nc']" - }, - { - "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#generate-the-kerchunk-reference-files.", - "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#generate-the-kerchunk-reference-files.", - "title": "PO.DAAC ECCO SSH", - "section": "Generate the Kerchunk reference files.", - "text": "Generate the Kerchunk reference files.\nDefine a function to generate the Kerchunk reference files. These files can take a little time to generate.\n\ndef gen_json(u):\n so = dict(\n mode= \"rb\", \n anon= False, \n default_fill_cache= False,\n default_cache_type= \"none\"\n )\n with fs.open(u, **so) as infile:\n h5chunks = SingleHdf5ToZarr(infile, u, inline_threshold=300)\n with open(f\"jsons/{u.split('/')[-1]}.json\", 'wb') as outf:\n outf.write(ujson.dumps(h5chunks.translate()).encode())\n\nCreate output jsons directory if one does not exist.\n\npathlib.Path('./jsons/').mkdir(exist_ok=True)\n\nUse the Dask Delayed function to create the Kerchunk reference file for each URL from the list of URLs in parallel\n\n%%time\n\nreference_files = []\nfor url in urls:\n ref = dask.delayed(gen_json)(url)\n reference_files.append(ref)\n\nreference_files_compute = dask.compute(*reference_files)\n\nCPU times: user 195 ms, sys: 83.4 ms, total: 278 ms\nWall time: 1.38 s\n\n\n\nfs_ref_list = fsspec.filesystem('file')\n\n\nreference_list = sorted([x for x in fs_ref_list.ls('jsons') if '.json' in x])\nreference_list\n\n['/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2014-12_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-01_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-02_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-03_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-04_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-05_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-06_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-07_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-08_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-09_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-10_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-11_ECCO_V4r4_latlon_0p50deg.nc.json',\n '/home/jovyan/earthdata-cloud-cookbook/examples/PODAAC/jsons/SEA_SURFACE_HEIGHT_mon_mean_2015-12_ECCO_V4r4_latlon_0p50deg.nc.json']" - }, - { - "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#read-single-netcdf4-using-kerchunk-reference-file", - "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#read-single-netcdf4-using-kerchunk-reference-file", - "title": "PO.DAAC ECCO SSH", - "section": "Read single netCDF4 using Kerchunk reference file", - "text": "Read single netCDF4 using Kerchunk reference file\nOpen the first reference file to read into an xarray dataset\n\nwith open(reference_list[0]) as j:\n reference = ujson.load(j)\n\nSet configurations options\n\ns_opts = {'skip_instance_cache':True} #json\nr_opts = {'anon':False, \n 'key':temp_creds_req['accessKeyId'], \n 'secret':temp_creds_req['secretAccessKey'], \n 'token':temp_creds_req['sessionToken']} #ncfiles\n\n\nfs_single = fsspec.filesystem(\"reference\",\n fo=reference,\n ref_storage_args=s_opts,\n remote_protocol='s3', \n remote_options=r_opts)\n\nRead in a single reference object. We get a lot of SerializationWarnings which are ignored here using the warning package.\nNOTE, the fill value, data range, min value, and max value may not match the source file. Will need to look into this more.\n\n%%time\n\nm = fs_single.get_mapper(\"\")\nds_single = xr.open_dataset(m, engine=\"zarr\", backend_kwargs={'consolidated':False}, chunks={})\nds_single\n\nCPU times: user 56.3 ms, sys: 26 ms, total: 82.2 ms\nWall time: 221 ms\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (time: 1, latitude: 360, longitude: 720, nv: 2)\nCoordinates:\n * latitude (latitude) float32 -89.75 -89.25 -88.75 ... 89.25 89.75\n latitude_bnds (latitude, nv) float32 dask.array<chunksize=(360, 2), meta=np.ndarray>\n * longitude (longitude) float32 -179.8 -179.2 -178.8 ... 179.2 179.8\n longitude_bnds (longitude, nv) float32 dask.array<chunksize=(720, 2), meta=np.ndarray>\n * time (time) datetime64[ns] 2014-12-16T12:00:00\n time_bnds (time, nv) datetime64[ns] dask.array<chunksize=(1, 2), meta=np.ndarray>\nDimensions without coordinates: nv\nData variables:\n SSH (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n SSHIBC (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n SSHNOIBC (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\nAttributes: (12/57)\n Conventions: CF-1.8, ACDD-1.3\n acknowledgement: This research was carried out by the Jet Pr...\n author: Ian Fenty and Ou Wang\n cdm_data_type: Grid\n comment: Fields provided on a regular lat-lon grid. ...\n coordinates_comment: Note: the global 'coordinates' attribute de...\n ... ...\n time_coverage_duration: P1M\n time_coverage_end: 2015-01-01T00:00:00\n time_coverage_resolution: P1M\n time_coverage_start: 2014-12-01T00:00:00\n title: ECCO Sea Surface Height - Monthly Mean 0.5 ...\n uuid: 08a2fc68-4158-11eb-b498-0cc47a3f6943xarray.DatasetDimensions:time: 1latitude: 360longitude: 720nv: 2Coordinates: (6)latitude(latitude)float32-89.75 -89.25 ... 89.25 89.75axis :Ybounds :latitude_bndscomment :uniform grid spacing from -89.75 to 89.75 by 0.5coverage_content_type :coordinatelong_name :latitude at grid cell centerstandard_name :latitudeunits :degrees_northarray([-89.75, -89.25, -88.75, ..., 88.75, 89.25, 89.75], dtype=float32)latitude_bnds(latitude, nv)float32dask.array<chunksize=(360, 2), meta=np.ndarray>coverage_content_type :coordinatelong_name :latitude bounds grid cells\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n2.81 kiB\n2.81 kiB\n\n\nShape\n(360, 2)\n(360, 2)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nlongitude\n\n\n(longitude)\n\n\nfloat32\n\n\n-179.8 -179.2 ... 179.2 179.8\n\n\n\n\naxis :\n\nX\n\nbounds :\n\nlongitude_bnds\n\ncomment :\n\nuniform grid spacing from -179.75 to 179.75 by 0.5\n\ncoverage_content_type :\n\ncoordinate\n\nlong_name :\n\nlongitude at grid cell center\n\nstandard_name :\n\nlongitude\n\nunits :\n\ndegrees_east\n\n\n\n\narray([-179.75, -179.25, -178.75, ..., 178.75, 179.25, 179.75],\n dtype=float32)\n\n\nlongitude_bnds\n\n\n(longitude, nv)\n\n\nfloat32\n\n\ndask.array<chunksize=(720, 2), meta=np.ndarray>\n\n\n\n\ncoverage_content_type :\n\ncoordinate\n\nlong_name :\n\nlongitude bounds grid cells\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n5.62 kiB\n5.62 kiB\n\n\nShape\n(720, 2)\n(720, 2)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\ntime\n\n\n(time)\n\n\ndatetime64[ns]\n\n\n2014-12-16T12:00:00\n\n\n\n\naxis :\n\nT\n\nbounds :\n\ntime_bnds\n\ncoverage_content_type :\n\ncoordinate\n\nlong_name :\n\ncenter time of averaging period\n\nstandard_name :\n\ntime\n\n\n\n\narray(['2014-12-16T12:00:00.000000000'], dtype='datetime64[ns]')\n\n\ntime_bnds\n\n\n(time, nv)\n\n\ndatetime64[ns]\n\n\ndask.array<chunksize=(1, 2), meta=np.ndarray>\n\n\n\n\ncomment :\n\nStart and end times of averaging period.\n\ncoverage_content_type :\n\ncoordinate\n\nlong_name :\n\ntime bounds of averaging period\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n16 B\n16 B\n\n\nShape\n(1, 2)\n(1, 2)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\ndatetime64[ns]\nnumpy.ndarray\n\n\n\n\n\n\n\n\nData variables: (3)\n\n\n\n\n\nSSH\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncomment :\n\nDynamic sea surface height anomaly above the geoid, suitable for comparisons with altimetry sea surface height data products that apply the inverse barometer (IB) correction. Note: SSH is calculated by correcting model sea level anomaly ETAN for three effects: a) global mean steric sea level changes related to density changes in the Boussinesq volume-conserving model (Greatbatch correction, see sterGloH), b) the inverted barometer (IB) effect (see SSHIBC) and c) sea level displacement due to sea-ice and snow pressure loading (see sIceLoad). SSH can be compared with the similarly-named SSH variable in previous ECCO products that did not include atmospheric pressure loading (e.g., Version 4 Release 3). Use SSHNOIBC for comparisons with altimetry data products that do NOT apply the IB correction.\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nDynamic sea surface height anomaly\n\nstandard_name :\n\nsea_surface_height_above_geoid\n\nunits :\n\nm\n\nvalid_max :\n\n1.4207719564437866\n\nvalid_min :\n\n-1.8805772066116333\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.99 MiB\n0.99 MiB\n\n\nShape\n(1, 360, 720)\n(1, 360, 720)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSSHIBC\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncomment :\n\nNot an SSH itself, but a correction to model sea level anomaly (ETAN) required to account for the static part of sea surface displacement by atmosphere pressure loading: SSH = SSHNOIBC - SSHIBC. Note: Use SSH for model-data comparisons with altimetry data products that DO apply the IB correction and SSHNOIBC for comparisons with altimetry data products that do NOT apply the IB correction.\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nThe inverted barometer (IB) correction to sea surface height due to atmospheric pressure loading\n\nunits :\n\nm\n\nvalid_max :\n\n0.5245633721351624\n\nvalid_min :\n\n-0.30144819617271423\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.99 MiB\n0.99 MiB\n\n\nShape\n(1, 360, 720)\n(1, 360, 720)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSSHNOIBC\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncomment :\n\nSea surface height anomaly above the geoid without the inverse barometer (IB) correction, suitable for comparisons with altimetry sea surface height data products that do NOT apply the inverse barometer (IB) correction. Note: SSHNOIBC is calculated by correcting model sea level anomaly ETAN for two effects: a) global mean steric sea level changes related to density changes in the Boussinesq volume-conserving model (Greatbatch correction, see sterGloH), b) sea level displacement due to sea-ice and snow pressure loading (see sIceLoad). In ECCO Version 4 Release 4 the model is forced with atmospheric pressure loading. SSHNOIBC does not correct for the static part of the effect of atmosphere pressure loading on sea surface height (the so-called inverse barometer (IB) correction). Use SSH for comparisons with altimetry data products that DO apply the IB correction.\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nSea surface height anomaly without the inverted barometer (IB) correction\n\nunits :\n\nm\n\nvalid_max :\n\n1.4550364017486572\n\nvalid_min :\n\n-1.6654272079467773\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.99 MiB\n0.99 MiB\n\n\nShape\n(1, 360, 720)\n(1, 360, 720)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nAttributes: (57)Conventions :CF-1.8, ACDD-1.3acknowledgement :This research was carried out by the Jet Propulsion Laboratory, managed by the California Institute of Technology under a contract with the National Aeronautics and Space Administration.author :Ian Fenty and Ou Wangcdm_data_type :Gridcomment :Fields provided on a regular lat-lon grid. They have been mapped to the regular lat-lon grid from the original ECCO lat-lon-cap 90 (llc90) native model grid. SSH (dynamic sea surface height) = SSHNOIBC (dynamic sea surface without the inverse barometer correction) - SSHIBC (inverse barometer correction). The inverted barometer correction accounts for variations in sea surface height due to atmospheric pressure variations.coordinates_comment :Note: the global 'coordinates' attribute describes auxillary coordinates.creator_email :ecco-group@mit.educreator_institution :NASA Jet Propulsion Laboratory (JPL)creator_name :ECCO Consortiumcreator_type :groupcreator_url :https://ecco-group.orgdate_created :2020-12-18T09:39:51date_issued :2020-12-18T09:39:51date_metadata_modified :2021-03-15T22:07:49date_modified :2021-03-15T22:07:49geospatial_bounds_crs :EPSG:4326geospatial_lat_max :90.0geospatial_lat_min :-90.0geospatial_lat_resolution :0.5geospatial_lat_units :degrees_northgeospatial_lon_max :180.0geospatial_lon_min :-180.0geospatial_lon_resolution :0.5geospatial_lon_units :degrees_easthistory :Inaugural release of an ECCO Central Estimate solution to PO.DAACid :10.5067/ECG5M-SSH44institution :NASA Jet Propulsion Laboratory (JPL)instrument_vocabulary :GCMD instrument keywordskeywords :EARTH SCIENCE > OCEANS > SEA SURFACE TOPOGRAPHY > SEA SURFACE HEIGHT, EARTH SCIENCE SERVICES > MODELS > EARTH SCIENCE REANALYSES/ASSIMILATION MODELSkeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordslicense :Public Domainmetadata_link :https://cmr.earthdata.nasa.gov/search/collections.umm_json?ShortName=ECCO_L4_SSH_05DEG_MONTHLY_V4R4naming_authority :gov.nasa.jplplatform :ERS-1/2, TOPEX/Poseidon, Geosat Follow-On (GFO), ENVISAT, Jason-1, Jason-2, CryoSat-2, SARAL/AltiKa, Jason-3, AVHRR, Aquarius, SSM/I, SSMIS, GRACE, DTU17MDT, Argo, WOCE, GO-SHIP, MEOP, Ice Tethered Profilers (ITP)platform_vocabulary :GCMD platform keywordsprocessing_level :L4product_name :SEA_SURFACE_HEIGHT_mon_mean_2014-12_ECCO_V4r4_latlon_0p50deg.ncproduct_time_coverage_end :2018-01-01T00:00:00product_time_coverage_start :1992-01-01T12:00:00product_version :Version 4, Release 4program :NASA Physical Oceanography, Cryosphere, Modeling, Analysis, and Prediction (MAP)project :Estimating the Circulation and Climate of the Ocean (ECCO)publisher_email :podaac@podaac.jpl.nasa.govpublisher_institution :PO.DAACpublisher_name :Physical Oceanography Distributed Active Archive Center (PO.DAAC)publisher_type :institutionpublisher_url :https://podaac.jpl.nasa.govreferences :ECCO Consortium, Fukumori, I., Wang, O., Fenty, I., Forget, G., Heimbach, P., & Ponte, R. M. 2020. Synopsis of the ECCO Central Production Global Ocean and Sea-Ice State Estimate (Version 4 Release 4). doi:10.5281/zenodo.3765928source :The ECCO V4r4 state estimate was produced by fitting a free-running solution of the MITgcm (checkpoint 66g) to satellite and in situ observational data in a least squares sense using the adjoint methodstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventionsummary :This dataset provides monthly-averaged dynamic sea surface height interpolated to a regular 0.5-degree grid from the ECCO Version 4 Release 4 (V4r4) ocean and sea-ice state estimate. Estimating the Circulation and Climate of the Ocean (ECCO) state estimates are dynamically and kinematically-consistent reconstructions of the three-dimensional, time-evolving ocean, sea-ice, and surface atmospheric states. ECCO V4r4 is a free-running solution of a global, nominally 1-degree configuration of the MIT general circulation model (MITgcm) that has been fit to observations in a least-squares sense. Observational data constraints used in V4r4 include sea surface height (SSH) from satellite altimeters [ERS-1/2, TOPEX/Poseidon, GFO, ENVISAT, Jason-1,2,3, CryoSat-2, and SARAL/AltiKa]; sea surface temperature (SST) from satellite radiometers [AVHRR], sea surface salinity (SSS) from the Aquarius satellite radiometer/scatterometer, ocean bottom pressure (OBP) from the GRACE satellite gravimeter; sea-ice concentration from satellite radiometers [SSM/I and SSMIS], and in-situ ocean temperature and salinity measured with conductivity-temperature-depth (CTD) sensors and expendable bathythermographs (XBTs) from several programs [e.g., WOCE, GO-SHIP, Argo, and others] and platforms [e.g., research vessels, gliders, moorings, ice-tethered profilers, and instrumented pinnipeds]. V4r4 covers the period 1992-01-01T12:00:00 to 2018-01-01T00:00:00.time_coverage_duration :P1Mtime_coverage_end :2015-01-01T00:00:00time_coverage_resolution :P1Mtime_coverage_start :2014-12-01T00:00:00title :ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4)uuid :08a2fc68-4158-11eb-b498-0cc47a3f6943" - }, - { - "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#read-multiple-netcdf4-files-using-kerchunk-reference-file", - "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#read-multiple-netcdf4-files-using-kerchunk-reference-file", - "title": "PO.DAAC ECCO SSH", - "section": "Read multiple netCDF4 files using Kerchunk reference file", - "text": "Read multiple netCDF4 files using Kerchunk reference file\nCombine the individual reference files into a single time series reference object\n\n%%time\n\nds_k =[]\nfor ref in reference_list:\n s_opts = s_opts\n r_opts = r_opts\n fs = fsspec.filesystem(\"reference\",\n fo=ref,\n ref_storage_args=s_opts,\n remote_protocol='s3',\n remote_options=r_opts)\n m = fs.get_mapper(\"\")\n ds_k.append(xr.open_dataset(m, engine=\"zarr\", backend_kwargs={'consolidated':False}, chunks={}))\n \nds_multi = xr.concat(ds_k, dim='time')\n \nds_multi\n\nCPU times: user 735 ms, sys: 31.4 ms, total: 766 ms\nWall time: 3.57 s\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (time: 13, latitude: 360, longitude: 720, nv: 2)\nCoordinates:\n * latitude (latitude) float32 -89.75 -89.25 -88.75 ... 89.25 89.75\n latitude_bnds (latitude, nv) float32 -90.0 -89.5 -89.5 ... 89.5 89.5 90.0\n * longitude (longitude) float32 -179.8 -179.2 -178.8 ... 179.2 179.8\n longitude_bnds (longitude, nv) float32 -180.0 -179.5 -179.5 ... 179.5 180.0\n * time (time) datetime64[ns] 2014-12-16T12:00:00 ... 2015-12-16T...\n time_bnds (time, nv) datetime64[ns] dask.array<chunksize=(1, 2), meta=np.ndarray>\nDimensions without coordinates: nv\nData variables:\n SSH (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n SSHIBC (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n SSHNOIBC (time, latitude, longitude) float32 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\nAttributes: (12/57)\n Conventions: CF-1.8, ACDD-1.3\n acknowledgement: This research was carried out by the Jet Pr...\n author: Ian Fenty and Ou Wang\n cdm_data_type: Grid\n comment: Fields provided on a regular lat-lon grid. ...\n coordinates_comment: Note: the global 'coordinates' attribute de...\n ... ...\n time_coverage_duration: P1M\n time_coverage_end: 2015-01-01T00:00:00\n time_coverage_resolution: P1M\n time_coverage_start: 2014-12-01T00:00:00\n title: ECCO Sea Surface Height - Monthly Mean 0.5 ...\n uuid: 08a2fc68-4158-11eb-b498-0cc47a3f6943xarray.DatasetDimensions:time: 13latitude: 360longitude: 720nv: 2Coordinates: (6)latitude(latitude)float32-89.75 -89.25 ... 89.25 89.75axis :Ybounds :latitude_bndscomment :uniform grid spacing from -89.75 to 89.75 by 0.5coverage_content_type :coordinatelong_name :latitude at grid cell centerstandard_name :latitudeunits :degrees_northarray([-89.75, -89.25, -88.75, ..., 88.75, 89.25, 89.75], dtype=float32)latitude_bnds(latitude, nv)float32-90.0 -89.5 -89.5 ... 89.5 90.0coverage_content_type :coordinatelong_name :latitude bounds grid cellsarray([[-90. , -89.5],\n [-89.5, -89. ],\n [-89. , -88.5],\n [-88.5, -88. ],\n [-88. , -87.5],\n [-87.5, -87. ],\n [-87. , -86.5],\n [-86.5, -86. ],\n [-86. , -85.5],\n [-85.5, -85. ],\n [-85. , -84.5],\n [-84.5, -84. ],\n [-84. , -83.5],\n [-83.5, -83. ],\n [-83. , -82.5],\n [-82.5, -82. ],\n [-82. , -81.5],\n [-81.5, -81. ],\n [-81. , -80.5],\n [-80.5, -80. ],\n...\n [ 80. , 80.5],\n [ 80.5, 81. ],\n [ 81. , 81.5],\n [ 81.5, 82. ],\n [ 82. , 82.5],\n [ 82.5, 83. ],\n [ 83. , 83.5],\n [ 83.5, 84. ],\n [ 84. , 84.5],\n [ 84.5, 85. ],\n [ 85. , 85.5],\n [ 85.5, 86. ],\n [ 86. , 86.5],\n [ 86.5, 87. ],\n [ 87. , 87.5],\n [ 87.5, 88. ],\n [ 88. , 88.5],\n [ 88.5, 89. ],\n [ 89. , 89.5],\n [ 89.5, 90. ]], dtype=float32)longitude(longitude)float32-179.8 -179.2 ... 179.2 179.8axis :Xbounds :longitude_bndscomment :uniform grid spacing from -179.75 to 179.75 by 0.5coverage_content_type :coordinatelong_name :longitude at grid cell centerstandard_name :longitudeunits :degrees_eastarray([-179.75, -179.25, -178.75, ..., 178.75, 179.25, 179.75],\n dtype=float32)longitude_bnds(longitude, nv)float32-180.0 -179.5 ... 179.5 180.0coverage_content_type :coordinatelong_name :longitude bounds grid cellsarray([[-180. , -179.5],\n [-179.5, -179. ],\n [-179. , -178.5],\n ...,\n [ 178.5, 179. ],\n [ 179. , 179.5],\n [ 179.5, 180. ]], dtype=float32)time(time)datetime64[ns]2014-12-16T12:00:00 ... 2015-12-...axis :Tbounds :time_bndscoverage_content_type :coordinatelong_name :center time of averaging periodstandard_name :timearray(['2014-12-16T12:00:00.000000000', '2015-01-16T12:00:00.000000000',\n '2015-02-15T00:00:00.000000000', '2015-03-16T12:00:00.000000000',\n '2015-04-16T00:00:00.000000000', '2015-05-16T12:00:00.000000000',\n '2015-06-16T00:00:00.000000000', '2015-07-16T12:00:00.000000000',\n '2015-08-16T12:00:00.000000000', '2015-09-16T00:00:00.000000000',\n '2015-10-16T12:00:00.000000000', '2015-11-16T00:00:00.000000000',\n '2015-12-16T12:00:00.000000000'], dtype='datetime64[ns]')time_bnds(time, nv)datetime64[ns]dask.array<chunksize=(1, 2), meta=np.ndarray>comment :Start and end times of averaging period.coverage_content_type :coordinatelong_name :time bounds of averaging period\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n208 B\n16 B\n\n\nShape\n(13, 2)\n(1, 2)\n\n\nCount\n39 Tasks\n13 Chunks\n\n\nType\ndatetime64[ns]\nnumpy.ndarray\n\n\n\n\n\n\n\nData variables: (3)\n\n\n\n\n\nSSH\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncomment :\n\nDynamic sea surface height anomaly above the geoid, suitable for comparisons with altimetry sea surface height data products that apply the inverse barometer (IB) correction. Note: SSH is calculated by correcting model sea level anomaly ETAN for three effects: a) global mean steric sea level changes related to density changes in the Boussinesq volume-conserving model (Greatbatch correction, see sterGloH), b) the inverted barometer (IB) effect (see SSHIBC) and c) sea level displacement due to sea-ice and snow pressure loading (see sIceLoad). SSH can be compared with the similarly-named SSH variable in previous ECCO products that did not include atmospheric pressure loading (e.g., Version 4 Release 3). Use SSHNOIBC for comparisons with altimetry data products that do NOT apply the IB correction.\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nDynamic sea surface height anomaly\n\nstandard_name :\n\nsea_surface_height_above_geoid\n\nunits :\n\nm\n\nvalid_max :\n\n1.4207719564437866\n\nvalid_min :\n\n-1.8805772066116333\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n12.85 MiB\n0.99 MiB\n\n\nShape\n(13, 360, 720)\n(1, 360, 720)\n\n\nCount\n39 Tasks\n13 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSSHIBC\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncomment :\n\nNot an SSH itself, but a correction to model sea level anomaly (ETAN) required to account for the static part of sea surface displacement by atmosphere pressure loading: SSH = SSHNOIBC - SSHIBC. Note: Use SSH for model-data comparisons with altimetry data products that DO apply the IB correction and SSHNOIBC for comparisons with altimetry data products that do NOT apply the IB correction.\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nThe inverted barometer (IB) correction to sea surface height due to atmospheric pressure loading\n\nunits :\n\nm\n\nvalid_max :\n\n0.5245633721351624\n\nvalid_min :\n\n-0.30144819617271423\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n12.85 MiB\n0.99 MiB\n\n\nShape\n(13, 360, 720)\n(1, 360, 720)\n\n\nCount\n39 Tasks\n13 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSSHNOIBC\n\n\n(time, latitude, longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\ncomment :\n\nSea surface height anomaly above the geoid without the inverse barometer (IB) correction, suitable for comparisons with altimetry sea surface height data products that do NOT apply the inverse barometer (IB) correction. Note: SSHNOIBC is calculated by correcting model sea level anomaly ETAN for two effects: a) global mean steric sea level changes related to density changes in the Boussinesq volume-conserving model (Greatbatch correction, see sterGloH), b) sea level displacement due to sea-ice and snow pressure loading (see sIceLoad). In ECCO Version 4 Release 4 the model is forced with atmospheric pressure loading. SSHNOIBC does not correct for the static part of the effect of atmosphere pressure loading on sea surface height (the so-called inverse barometer (IB) correction). Use SSH for comparisons with altimetry data products that DO apply the IB correction.\n\ncoverage_content_type :\n\nmodelResult\n\nlong_name :\n\nSea surface height anomaly without the inverted barometer (IB) correction\n\nunits :\n\nm\n\nvalid_max :\n\n1.4550364017486572\n\nvalid_min :\n\n-1.6654272079467773\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n12.85 MiB\n0.99 MiB\n\n\nShape\n(13, 360, 720)\n(1, 360, 720)\n\n\nCount\n39 Tasks\n13 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nAttributes: (57)Conventions :CF-1.8, ACDD-1.3acknowledgement :This research was carried out by the Jet Propulsion Laboratory, managed by the California Institute of Technology under a contract with the National Aeronautics and Space Administration.author :Ian Fenty and Ou Wangcdm_data_type :Gridcomment :Fields provided on a regular lat-lon grid. They have been mapped to the regular lat-lon grid from the original ECCO lat-lon-cap 90 (llc90) native model grid. SSH (dynamic sea surface height) = SSHNOIBC (dynamic sea surface without the inverse barometer correction) - SSHIBC (inverse barometer correction). The inverted barometer correction accounts for variations in sea surface height due to atmospheric pressure variations.coordinates_comment :Note: the global 'coordinates' attribute describes auxillary coordinates.creator_email :ecco-group@mit.educreator_institution :NASA Jet Propulsion Laboratory (JPL)creator_name :ECCO Consortiumcreator_type :groupcreator_url :https://ecco-group.orgdate_created :2020-12-18T09:39:51date_issued :2020-12-18T09:39:51date_metadata_modified :2021-03-15T22:07:49date_modified :2021-03-15T22:07:49geospatial_bounds_crs :EPSG:4326geospatial_lat_max :90.0geospatial_lat_min :-90.0geospatial_lat_resolution :0.5geospatial_lat_units :degrees_northgeospatial_lon_max :180.0geospatial_lon_min :-180.0geospatial_lon_resolution :0.5geospatial_lon_units :degrees_easthistory :Inaugural release of an ECCO Central Estimate solution to PO.DAACid :10.5067/ECG5M-SSH44institution :NASA Jet Propulsion Laboratory (JPL)instrument_vocabulary :GCMD instrument keywordskeywords :EARTH SCIENCE > OCEANS > SEA SURFACE TOPOGRAPHY > SEA SURFACE HEIGHT, EARTH SCIENCE SERVICES > MODELS > EARTH SCIENCE REANALYSES/ASSIMILATION MODELSkeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordslicense :Public Domainmetadata_link :https://cmr.earthdata.nasa.gov/search/collections.umm_json?ShortName=ECCO_L4_SSH_05DEG_MONTHLY_V4R4naming_authority :gov.nasa.jplplatform :ERS-1/2, TOPEX/Poseidon, Geosat Follow-On (GFO), ENVISAT, Jason-1, Jason-2, CryoSat-2, SARAL/AltiKa, Jason-3, AVHRR, Aquarius, SSM/I, SSMIS, GRACE, DTU17MDT, Argo, WOCE, GO-SHIP, MEOP, Ice Tethered Profilers (ITP)platform_vocabulary :GCMD platform keywordsprocessing_level :L4product_name :SEA_SURFACE_HEIGHT_mon_mean_2014-12_ECCO_V4r4_latlon_0p50deg.ncproduct_time_coverage_end :2018-01-01T00:00:00product_time_coverage_start :1992-01-01T12:00:00product_version :Version 4, Release 4program :NASA Physical Oceanography, Cryosphere, Modeling, Analysis, and Prediction (MAP)project :Estimating the Circulation and Climate of the Ocean (ECCO)publisher_email :podaac@podaac.jpl.nasa.govpublisher_institution :PO.DAACpublisher_name :Physical Oceanography Distributed Active Archive Center (PO.DAAC)publisher_type :institutionpublisher_url :https://podaac.jpl.nasa.govreferences :ECCO Consortium, Fukumori, I., Wang, O., Fenty, I., Forget, G., Heimbach, P., & Ponte, R. M. 2020. Synopsis of the ECCO Central Production Global Ocean and Sea-Ice State Estimate (Version 4 Release 4). doi:10.5281/zenodo.3765928source :The ECCO V4r4 state estimate was produced by fitting a free-running solution of the MITgcm (checkpoint 66g) to satellite and in situ observational data in a least squares sense using the adjoint methodstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventionsummary :This dataset provides monthly-averaged dynamic sea surface height interpolated to a regular 0.5-degree grid from the ECCO Version 4 Release 4 (V4r4) ocean and sea-ice state estimate. Estimating the Circulation and Climate of the Ocean (ECCO) state estimates are dynamically and kinematically-consistent reconstructions of the three-dimensional, time-evolving ocean, sea-ice, and surface atmospheric states. ECCO V4r4 is a free-running solution of a global, nominally 1-degree configuration of the MIT general circulation model (MITgcm) that has been fit to observations in a least-squares sense. Observational data constraints used in V4r4 include sea surface height (SSH) from satellite altimeters [ERS-1/2, TOPEX/Poseidon, GFO, ENVISAT, Jason-1,2,3, CryoSat-2, and SARAL/AltiKa]; sea surface temperature (SST) from satellite radiometers [AVHRR], sea surface salinity (SSS) from the Aquarius satellite radiometer/scatterometer, ocean bottom pressure (OBP) from the GRACE satellite gravimeter; sea-ice concentration from satellite radiometers [SSM/I and SSMIS], and in-situ ocean temperature and salinity measured with conductivity-temperature-depth (CTD) sensors and expendable bathythermographs (XBTs) from several programs [e.g., WOCE, GO-SHIP, Argo, and others] and platforms [e.g., research vessels, gliders, moorings, ice-tethered profilers, and instrumented pinnipeds]. V4r4 covers the period 1992-01-01T12:00:00 to 2018-01-01T00:00:00.time_coverage_duration :P1Mtime_coverage_end :2015-01-01T00:00:00time_coverage_resolution :P1Mtime_coverage_start :2014-12-01T00:00:00title :ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4)uuid :08a2fc68-4158-11eb-b498-0cc47a3f6943\n\n\n\nds_multi['SSH']\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray 'SSH' (time: 13, latitude: 360, longitude: 720)>\ndask.array<concatenate, shape=(13, 360, 720), dtype=float32, chunksize=(1, 360, 720), chunktype=numpy.ndarray>\nCoordinates:\n * latitude (latitude) float32 -89.75 -89.25 -88.75 ... 88.75 89.25 89.75\n * longitude (longitude) float32 -179.8 -179.2 -178.8 ... 178.8 179.2 179.8\n * time (time) datetime64[ns] 2014-12-16T12:00:00 ... 2015-12-16T12:00:00\nAttributes:\n comment: Dynamic sea surface height anomaly above the geoi...\n coverage_content_type: modelResult\n long_name: Dynamic sea surface height anomaly\n standard_name: sea_surface_height_above_geoid\n units: m\n valid_max: 1.4207719564437866\n valid_min: -1.8805772066116333xarray.DataArray'SSH'time: 13latitude: 360longitude: 720dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n12.85 MiB\n0.99 MiB\n\n\nShape\n(13, 360, 720)\n(1, 360, 720)\n\n\nCount\n39 Tasks\n13 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nCoordinates: (3)latitude(latitude)float32-89.75 -89.25 ... 89.25 89.75axis :Ybounds :latitude_bndscomment :uniform grid spacing from -89.75 to 89.75 by 0.5coverage_content_type :coordinatelong_name :latitude at grid cell centerstandard_name :latitudeunits :degrees_northarray([-89.75, -89.25, -88.75, ..., 88.75, 89.25, 89.75], dtype=float32)longitude(longitude)float32-179.8 -179.2 ... 179.2 179.8axis :Xbounds :longitude_bndscomment :uniform grid spacing from -179.75 to 179.75 by 0.5coverage_content_type :coordinatelong_name :longitude at grid cell centerstandard_name :longitudeunits :degrees_eastarray([-179.75, -179.25, -178.75, ..., 178.75, 179.25, 179.75],\n dtype=float32)time(time)datetime64[ns]2014-12-16T12:00:00 ... 2015-12-...axis :Tbounds :time_bndscoverage_content_type :coordinatelong_name :center time of averaging periodstandard_name :timearray(['2014-12-16T12:00:00.000000000', '2015-01-16T12:00:00.000000000',\n '2015-02-15T00:00:00.000000000', '2015-03-16T12:00:00.000000000',\n '2015-04-16T00:00:00.000000000', '2015-05-16T12:00:00.000000000',\n '2015-06-16T00:00:00.000000000', '2015-07-16T12:00:00.000000000',\n '2015-08-16T12:00:00.000000000', '2015-09-16T00:00:00.000000000',\n '2015-10-16T12:00:00.000000000', '2015-11-16T00:00:00.000000000',\n '2015-12-16T12:00:00.000000000'], dtype='datetime64[ns]')Attributes: (7)comment :Dynamic sea surface height anomaly above the geoid, suitable for comparisons with altimetry sea surface height data products that apply the inverse barometer (IB) correction. Note: SSH is calculated by correcting model sea level anomaly ETAN for three effects: a) global mean steric sea level changes related to density changes in the Boussinesq volume-conserving model (Greatbatch correction, see sterGloH), b) the inverted barometer (IB) effect (see SSHIBC) and c) sea level displacement due to sea-ice and snow pressure loading (see sIceLoad). SSH can be compared with the similarly-named SSH variable in previous ECCO products that did not include atmospheric pressure loading (e.g., Version 4 Release 3). Use SSHNOIBC for comparisons with altimetry data products that do NOT apply the IB correction.coverage_content_type :modelResultlong_name :Dynamic sea surface height anomalystandard_name :sea_surface_height_above_geoidunits :mvalid_max :1.4207719564437866valid_min :-1.8805772066116333\n\n\n\n# Commenting for quarto site render\n# ds_multi['SSH'].hvplot.image()" - }, - { - "objectID": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#references", - "href": "examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html#references", - "title": "PO.DAAC ECCO SSH", - "section": "References", - "text": "References\n\nhttps://github.com/fsspec/kerchunk\nhttps://medium.com/pangeo/fake-it-until-you-make-it-reading-goes-netcdf4-data-on-aws-s3-as-zarr-for-rapid-data-access-61e33f8fe685\nhttps://medium.com/pangeo/cloud-performant-reading-of-netcdf4-hdf5-data-using-the-zarr-library-1a95c5c92314" - }, - { - "objectID": "examples/ORNL/Data_Access__Direct_S3_Access__ORNL_DAYMET.html#single-file-in-region-direct-s3-access-of-netcdf-file", - "href": "examples/ORNL/Data_Access__Direct_S3_Access__ORNL_DAYMET.html#single-file-in-region-direct-s3-access-of-netcdf-file", - "title": "Direct S3 Data Access: NetCDF - Daymet v4 Daily TMAX Example", - "section": "Single file in-region direct S3 access of netcdf file", - "text": "Single file in-region direct S3 access of netcdf file\n\nfs_s3 = s3fs.S3FileSystem(anon=False, key=temp_creds_req['accessKeyId'], secret=temp_creds_req['secretAccessKey'], token=temp_creds_req['sessionToken'])\n\n\ns3_url = s3_urls[0]\ns3_url\n\n's3://ornl-cumulus-prod-protected/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_tmax_2015.nc'\n\n\n\ns3_file_obj = fs_s3.open(s3_url, mode='rb')\n\n\nxr_ds = xr.open_dataset(s3_file_obj, chunks='auto', engine='h5netcdf')\nxr_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (x: 7814, y: 8075, time: 365, nv: 2)\nCoordinates:\n * x (x) float32 -4.56e+06 -4.559e+06 ... 3.253e+06\n * y (y) float32 4.984e+06 4.983e+06 ... -3.09e+06\n lat (y, x) float32 dask.array<chunksize=(8075, 3907), meta=np.ndarray>\n lon (y, x) float32 dask.array<chunksize=(8075, 3907), meta=np.ndarray>\n * time (time) datetime64[ns] 2015-01-01T12:00:00 ... 20...\nDimensions without coordinates: nv\nData variables:\n yearday (time) int16 dask.array<chunksize=(365,), meta=np.ndarray>\n time_bnds (time, nv) datetime64[ns] dask.array<chunksize=(365, 2), meta=np.ndarray>\n lambert_conformal_conic int16 ...\n tmax (time, y, x) float32 dask.array<chunksize=(55, 475, 1282), meta=np.ndarray>\nAttributes:\n start_year: 2015\n source: Daymet Software Version 4.0\n Version_software: Daymet Software Version 4.0\n Version_data: Daymet Data Version 4.0\n Conventions: CF-1.6\n citation: Please see http://daymet.ornl.gov/ for current Daymet ...\n references: Please see http://daymet.ornl.gov/ for current informa...xarray.DatasetDimensions:x: 7814y: 8075time: 365nv: 2Coordinates: (5)x(x)float32-4.56e+06 -4.559e+06 ... 3.253e+06units :mlong_name :x coordinate of projectionstandard_name :projection_x_coordinatearray([-4560250., -4559250., -4558250., ..., 3250750., 3251750., 3252750.],\n dtype=float32)y(y)float324.984e+06 4.983e+06 ... -3.09e+06units :mlong_name :y coordinate of projectionstandard_name :projection_y_coordinatearray([ 4984000., 4983000., 4982000., ..., -3088000., -3089000., -3090000.],\n dtype=float32)lat(y, x)float32dask.array<chunksize=(8075, 3907), meta=np.ndarray>units :degrees_northlong_name :latitude coordinatestandard_name :latitude\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n240.70 MiB\n120.35 MiB\n\n\nShape\n(8075, 7814)\n(8075, 3907)\n\n\nCount\n3 Tasks\n2 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nlon\n\n\n(y, x)\n\n\nfloat32\n\n\ndask.array<chunksize=(8075, 3907), meta=np.ndarray>\n\n\n\n\nunits :\n\ndegrees_east\n\nlong_name :\n\nlongitude coordinate\n\nstandard_name :\n\nlongitude\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n240.70 MiB\n120.35 MiB\n\n\nShape\n(8075, 7814)\n(8075, 3907)\n\n\nCount\n3 Tasks\n2 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\ntime\n\n\n(time)\n\n\ndatetime64[ns]\n\n\n2015-01-01T12:00:00 ... 2015-12-...\n\n\n\n\nstandard_name :\n\ntime\n\nbounds :\n\ntime_bnds\n\nlong_name :\n\n24-hour day based on local time\n\n\n\n\narray(['2015-01-01T12:00:00.000000000', '2015-01-02T12:00:00.000000000',\n '2015-01-03T12:00:00.000000000', ..., '2015-12-29T12:00:00.000000000',\n '2015-12-30T12:00:00.000000000', '2015-12-31T12:00:00.000000000'],\n dtype='datetime64[ns]')\n\nData variables: (4)\n\n\n\n\n\nyearday\n\n\n(time)\n\n\nint16\n\n\ndask.array<chunksize=(365,), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nday of year (DOY) starting with day 1 on Januaray 1st\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n730 B\n730 B\n\n\nShape\n(365,)\n(365,)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nint16\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\ntime_bnds\n\n\n(time, nv)\n\n\ndatetime64[ns]\n\n\ndask.array<chunksize=(365, 2), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n5.70 kiB\n5.70 kiB\n\n\nShape\n(365, 2)\n(365, 2)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\ndatetime64[ns]\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nlambert_conformal_conic\n\n\n()\n\n\nint16\n\n\n...\n\n\n\n\ngrid_mapping_name :\n\nlambert_conformal_conic\n\nlongitude_of_central_meridian :\n\n-100.0\n\nlatitude_of_projection_origin :\n\n42.5\n\nfalse_easting :\n\n0.0\n\nfalse_northing :\n\n0.0\n\nstandard_parallel :\n\n[25. 60.]\n\nsemi_major_axis :\n\n6378137.0\n\ninverse_flattening :\n\n298.257223563\n\n\n\n\narray(-32767, dtype=int16)\n\n\ntmax\n\n\n(time, y, x)\n\n\nfloat32\n\n\ndask.array<chunksize=(55, 475, 1282), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ndaily maximum temperature\n\nunits :\n\ndegrees C\n\ngrid_mapping :\n\nlambert_conformal_conic\n\ncell_methods :\n\narea: mean time: maximum\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n85.80 GiB\n127.76 MiB\n\n\nShape\n(365, 8075, 7814)\n(55, 475, 1282)\n\n\nCount\n834 Tasks\n833 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nAttributes: (7)start_year :2015source :Daymet Software Version 4.0Version_software :Daymet Software Version 4.0Version_data :Daymet Data Version 4.0Conventions :CF-1.6citation :Please see http://daymet.ornl.gov/ for current Daymet data citation informationreferences :Please see http://daymet.ornl.gov/ for current information on Daymet references" - }, - { - "objectID": "examples/ORNL/Data_Access__Direct_S3_Access__ORNL_DAYMET.html#multi-file-in-region-direct-s3-access-of-netcdf-files", - "href": "examples/ORNL/Data_Access__Direct_S3_Access__ORNL_DAYMET.html#multi-file-in-region-direct-s3-access-of-netcdf-files", - "title": "Direct S3 Data Access: NetCDF - Daymet v4 Daily TMAX Example", - "section": "Multi-file in-region direct S3 access of netcdf files", - "text": "Multi-file in-region direct S3 access of netcdf files\n\n# Iterate through remote_files to create a fileset\nfileset = [fs_s3.open(file) for file in s3_urls]\n\n\n# This works...if you rerun this line and get a context manager error, try 1. rerunning the line above then this line \nxr_ts = xr.open_mfdataset(fileset, chunks='auto', engine='h5netcdf')\n\n\nxr_ts\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (x: 7814, y: 8075, time: 2190, nv: 2)\nCoordinates:\n * x (x) float32 -4.56e+06 -4.559e+06 ... 3.253e+06\n * y (y) float32 4.984e+06 4.983e+06 ... -3.09e+06\n lat (y, x) float32 dask.array<chunksize=(8075, 3907), meta=np.ndarray>\n lon (y, x) float32 dask.array<chunksize=(8075, 3907), meta=np.ndarray>\n * time (time) datetime64[ns] 2015-01-01T12:00:00 ... 20...\nDimensions without coordinates: nv\nData variables:\n yearday (time) int16 dask.array<chunksize=(365,), meta=np.ndarray>\n time_bnds (time, nv) datetime64[ns] dask.array<chunksize=(365, 2), meta=np.ndarray>\n lambert_conformal_conic (time) int16 -32767 -32767 -32767 ... -32767 -32767\n tmax (time, y, x) float32 dask.array<chunksize=(55, 475, 1282), meta=np.ndarray>\nAttributes:\n start_year: 2015\n source: Daymet Software Version 4.0\n Version_software: Daymet Software Version 4.0\n Version_data: Daymet Data Version 4.0\n Conventions: CF-1.6\n citation: Please see http://daymet.ornl.gov/ for current Daymet ...\n references: Please see http://daymet.ornl.gov/ for current informa...xarray.DatasetDimensions:x: 7814y: 8075time: 2190nv: 2Coordinates: (5)x(x)float32-4.56e+06 -4.559e+06 ... 3.253e+06units :mlong_name :x coordinate of projectionstandard_name :projection_x_coordinatearray([-4560250., -4559250., -4558250., ..., 3250750., 3251750., 3252750.],\n dtype=float32)y(y)float324.984e+06 4.983e+06 ... -3.09e+06units :mlong_name :y coordinate of projectionstandard_name :projection_y_coordinatearray([ 4984000., 4983000., 4982000., ..., -3088000., -3089000., -3090000.],\n dtype=float32)lat(y, x)float32dask.array<chunksize=(8075, 3907), meta=np.ndarray>units :degrees_northlong_name :latitude coordinatestandard_name :latitude\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n240.70 MiB\n120.35 MiB\n\n\nShape\n(8075, 7814)\n(8075, 3907)\n\n\nCount\n51 Tasks\n2 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nlon\n\n\n(y, x)\n\n\nfloat32\n\n\ndask.array<chunksize=(8075, 3907), meta=np.ndarray>\n\n\n\n\nunits :\n\ndegrees_east\n\nlong_name :\n\nlongitude coordinate\n\nstandard_name :\n\nlongitude\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n240.70 MiB\n120.35 MiB\n\n\nShape\n(8075, 7814)\n(8075, 3907)\n\n\nCount\n51 Tasks\n2 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\ntime\n\n\n(time)\n\n\ndatetime64[ns]\n\n\n2015-01-01T12:00:00 ... 2020-12-...\n\n\n\n\nstandard_name :\n\ntime\n\nbounds :\n\ntime_bnds\n\nlong_name :\n\n24-hour day based on local time\n\n\n\n\narray(['2015-01-01T12:00:00.000000000', '2015-01-02T12:00:00.000000000',\n '2015-01-03T12:00:00.000000000', ..., '2020-12-28T12:00:00.000000000',\n '2020-12-29T12:00:00.000000000', '2020-12-30T12:00:00.000000000'],\n dtype='datetime64[ns]')\n\nData variables: (4)\n\n\n\n\n\nyearday\n\n\n(time)\n\n\nint16\n\n\ndask.array<chunksize=(365,), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nday of year (DOY) starting with day 1 on Januaray 1st\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n4.28 kiB\n730 B\n\n\nShape\n(2190,)\n(365,)\n\n\nCount\n18 Tasks\n6 Chunks\n\n\nType\nint16\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\ntime_bnds\n\n\n(time, nv)\n\n\ndatetime64[ns]\n\n\ndask.array<chunksize=(365, 2), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n34.22 kiB\n5.70 kiB\n\n\nShape\n(2190, 2)\n(365, 2)\n\n\nCount\n18 Tasks\n6 Chunks\n\n\nType\ndatetime64[ns]\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nlambert_conformal_conic\n\n\n(time)\n\n\nint16\n\n\n-32767 -32767 ... -32767 -32767\n\n\n\n\ngrid_mapping_name :\n\nlambert_conformal_conic\n\nlongitude_of_central_meridian :\n\n-100.0\n\nlatitude_of_projection_origin :\n\n42.5\n\nfalse_easting :\n\n0.0\n\nfalse_northing :\n\n0.0\n\nstandard_parallel :\n\n[25. 60.]\n\nsemi_major_axis :\n\n6378137.0\n\ninverse_flattening :\n\n298.257223563\n\n\n\n\narray([-32767, -32767, -32767, ..., -32767, -32767, -32767], dtype=int16)\n\n\ntmax\n\n\n(time, y, x)\n\n\nfloat32\n\n\ndask.array<chunksize=(55, 475, 1282), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ndaily maximum temperature\n\nunits :\n\ndegrees C\n\ngrid_mapping :\n\nlambert_conformal_conic\n\ncell_methods :\n\narea: mean time: maximum\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n514.78 GiB\n127.76 MiB\n\n\nShape\n(2190, 8075, 7814)\n(55, 475, 1282)\n\n\nCount\n10002 Tasks\n4998 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nAttributes: (7)start_year :2015source :Daymet Software Version 4.0Version_software :Daymet Software Version 4.0Version_data :Daymet Data Version 4.0Conventions :CF-1.6citation :Please see http://daymet.ornl.gov/ for current Daymet data citation informationreferences :Please see http://daymet.ornl.gov/ for current information on Daymet references\n\n\n\n#xr_ts.SSH.hvplot.image()" - }, - { - "objectID": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html", - "href": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html", - "title": "Searching and Downloading NSIDC Cloud Collections", - "section": "", - "text": "Programmatic access and processing of NSIDC data can happen in 2 ways, using the old Search -> Download -> Analize pattern or using a more modern Search -> Process_in_the_cloud -> Analyze approach.\nThere is nothing wrong with downloading data to our local machine but that can get complicated or even impossible if a dataset is too large. For this reason NSIDC along with other NASA data centers started to collocate or migrate their dataset holdings to the cloud.\nIn order to use NSIDC cloud collections we need to 1. Authenticate ourselves with the NASA Earthdata Login API (EDL). 2. Search granules/collections using a CMR client that supports authentication 3. Parse CMR responses looking for AWS S3 URLs 4. Access the data granules using temporary AWS credentials given by the NSIDC cloud credentials endpoint" - }, - { - "objectID": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#data-used", - "href": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#data-used", - "title": "Searching and Downloading NSIDC Cloud Collections", - "section": "Data used:", - "text": "Data used:\n\nICESat-2 ATL03: This data set contains height above the WGS 84 ellipsoid (ITRF2014 reference frame), latitude, longitude, and time for all photons." - }, - { - "objectID": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#requirements", - "href": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#requirements", - "title": "Searching and Downloading NSIDC Cloud Collections", - "section": "Requirements", - "text": "Requirements\n\nNASA Eartdata Login (EDL) credentials\npython libraries:\n\naws-cli\nxarray\nfs-s3fs\nzarr\ncmr\n\npython-cmr (optional fallback) NSIDC fork" - }, - { - "objectID": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#querying-cmr-for-nsidc-data", - "href": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#querying-cmr-for-nsidc-data", - "title": "Searching and Downloading NSIDC Cloud Collections", - "section": "Querying CMR for NSIDC data", - "text": "Querying CMR for NSIDC data\nMost collections at NSIDC have not being migrated to the cloud and can be found using CMR with no authentication at all. Here is a simple example for altimeter data (ATL03) coming from the ICESat-2 mission. First we’ll search the regular collection and then we’ll do the same using the cloud collection.\nNote: This notebook uses a low level CMR endpoint, this won’t be not the only workflow for data discovery.\n\nfrom cmr.search import collection as cmr_collection\nfrom cmr.search import granule \nfrom cmr.auth import token\n\nimport textwrap\n# NON_AWS collections are hosted at the NSIDC DAAC data center\n# AWS_CLOUD collections are hosted at AWS S3 us-west-2\nNSIDC_PROVIDERS = {\n 'NSIDC_HOSTED': 'NSIDC_ECS', \n 'AWS_HOSTED':'NSIDC_CPRD'\n}\n\n# First let's search for some collections hosted at NSIDC using a keyword\ncollections = cmr_collection.search({'keyword':'ice',\n 'provider': NSIDC_PROVIDERS['NSIDC_HOSTED']})\n\n# Let's print some information about the first 3 collection that match our provider\nfor collection in collections[0:3]:\n wrapped_abstract = '\\n'.join(textwrap.wrap(f\"Abstract: {collection['umm']['Abstract']}\", 80)) + '\\n'\n print(f\"concept-id: {collection['meta']['concept-id']}\\n\" +\n f\"Title: {collection['umm']['EntryTitle']}\\n\" +\n wrapped_abstract)\n\nconcept-id: C1997321091-NSIDC_ECS\nTitle: ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004\nAbstract: This data set (ATL03) contains height above the WGS 84 ellipsoid\n(ITRF2014 reference frame), latitude, longitude, and time for all photons\ndownlinked by the Advanced Topographic Laser Altimeter System (ATLAS) instrument\non board the Ice, Cloud and land Elevation Satellite-2 (ICESat-2) observatory.\nThe ATL03 product was designed to be a single source for all photon data and\nancillary information needed by higher-level ATLAS/ICESat-2 products. As such,\nit also includes spacecraft and instrument parameters and ancillary data not\nexplicitly required for ATL03.\n\nconcept-id: C1705401930-NSIDC_ECS\nTitle: ATLAS/ICESat-2 L2A Global Geolocated Photon Data V003\nAbstract: This data set (ATL03) contains height above the WGS 84 ellipsoid\n(ITRF2014 reference frame), latitude, longitude, and time for all photons\ndownlinked by the Advanced Topographic Laser Altimeter System (ATLAS) instrument\non board the Ice, Cloud and land Elevation Satellite-2 (ICESat-2) observatory.\nThe ATL03 product was designed to be a single source for all photon data and\nancillary information needed by higher-level ATLAS/ICESat-2 products. As such,\nit also includes spacecraft and instrument parameters and ancillary data not\nexplicitly required for ATL03.\n\nconcept-id: C2003771331-NSIDC_ECS\nTitle: ATLAS/ICESat-2 L3A Land Ice Height V004\nAbstract: This data set (ATL06) provides geolocated, land-ice surface heights\n(above the WGS 84 ellipsoid, ITRF2014 reference frame), plus ancillary\nparameters that can be used to interpret and assess the quality of the height\nestimates. The data were acquired by the Advanced Topographic Laser Altimeter\nSystem (ATLAS) instrument on board the Ice, Cloud and land Elevation Satellite-2\n(ICESat-2) observatory.\n\n\n\n\n# Now let's do the same with short names, a more specific way of finding data.\n\n#First let's search for some collections hosted at NSIDC\ncollections = cmr_collection.search({'short_name':'ATL03',\n 'provider': NSIDC_PROVIDERS['NSIDC_HOSTED']})\n\n# Note how we get back the same collection twice, that's because we have 2 versions available.\nfor collection in collections[0:3]:\n wrapped_abstract = '\\n'.join(textwrap.wrap(f\"Abstract: {collection['umm']['Abstract']}\", 80)) + '\\n'\n print(f\"concept-id: {collection['meta']['concept-id']}\\n\" +\n f\"Title: {collection['umm']['EntryTitle']}\\n\" +\n wrapped_abstract)\n\nconcept-id: C1997321091-NSIDC_ECS\nTitle: ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004\nAbstract: This data set (ATL03) contains height above the WGS 84 ellipsoid\n(ITRF2014 reference frame), latitude, longitude, and time for all photons\ndownlinked by the Advanced Topographic Laser Altimeter System (ATLAS) instrument\non board the Ice, Cloud and land Elevation Satellite-2 (ICESat-2) observatory.\nThe ATL03 product was designed to be a single source for all photon data and\nancillary information needed by higher-level ATLAS/ICESat-2 products. As such,\nit also includes spacecraft and instrument parameters and ancillary data not\nexplicitly required for ATL03.\n\nconcept-id: C1705401930-NSIDC_ECS\nTitle: ATLAS/ICESat-2 L2A Global Geolocated Photon Data V003\nAbstract: This data set (ATL03) contains height above the WGS 84 ellipsoid\n(ITRF2014 reference frame), latitude, longitude, and time for all photons\ndownlinked by the Advanced Topographic Laser Altimeter System (ATLAS) instrument\non board the Ice, Cloud and land Elevation Satellite-2 (ICESat-2) observatory.\nThe ATL03 product was designed to be a single source for all photon data and\nancillary information needed by higher-level ATLAS/ICESat-2 products. As such,\nit also includes spacecraft and instrument parameters and ancillary data not\nexplicitly required for ATL03.\n\n\n\n\n# now that we have the concept-ids we can look for data granules in that collection and pass spatiotemporal parameters.\nfrom cmr_serializer import QueryResult\n\n# a bbox over Juneau Icefield \n# bbox = min Longitude , min Latitude , max Longitude , max Latitude \nquery = {'concept-id': 'C1997321091-NSIDC_ECS',\n 'bounding_box': '-135.1977,58.3325,-133.3410,58.9839'}\n\n# Querying for ATL03 v3 using its concept-id and a bounding box\nresults = granule.search(query, limit=1000)\n# This is a wrapper with convenient methods to work with CMR query results.\ngranules = QueryResult(results).items()\n\nprint(f\"Total granules found: {len(results)} \\n\")\nfor g in granules[0:3]:\n display(g)\n\nTotal granules found: 201 \n\n\n\n\n \n Id: ATL03_20181014001049_02350102_004_01.h5\n Collection: {'EntryTitle': 'ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004'}\n Spatial coverage: {'HorizontalSpatialDomain': {'Orbit': {'AscendingCrossing': -127.0482205607256, 'StartLatitude': 27.0, 'StartDirection': 'A', 'EndLatitude': 59.5, 'EndDirection': 'A'}}}\n Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2018-10-14T00:10:49.722Z', 'EndingDateTime': '2018-10-14T00:19:19.918Z'}}\n Size(MB): 1764.5729866028 \n Data: https://n5eil01u.ecs.nsidc.org/DP9/ATLAS/ATL03.004/2018.10.14/ATL03_20181014001049_02350102_004_01.h5\n \n \n \n\n\n\n \n Id: ATL03_20181015124359_02580106_004_01.h5\n Collection: {'EntryTitle': 'ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004'}\n Spatial coverage: {'HorizontalSpatialDomain': {'Orbit': {'AscendingCrossing': 49.70324528818096, 'StartLatitude': 59.5, 'StartDirection': 'D', 'EndLatitude': 27.0, 'EndDirection': 'D'}}}\n Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2018-10-15T12:43:57.696Z', 'EndingDateTime': '2018-10-15T12:52:28.274Z'}}\n Size(MB): 276.2403841019 \n Data: https://n5eil01u.ecs.nsidc.org/DP9/ATLAS/ATL03.004/2018.10.15/ATL03_20181015124359_02580106_004_01.h5\n \n \n \n\n\n\n \n Id: ATL03_20181018000228_02960102_004_01.h5\n Collection: {'EntryTitle': 'ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004'}\n Spatial coverage: {'HorizontalSpatialDomain': {'Orbit': {'AscendingCrossing': -127.82682215638665, 'StartLatitude': 27.0, 'StartDirection': 'A', 'EndLatitude': 59.5, 'EndDirection': 'A'}}}\n Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2018-10-18T00:02:28.717Z', 'EndingDateTime': '2018-10-18T00:10:58.903Z'}}\n Size(MB): 877.0574979782 \n Data: https://n5eil01u.ecs.nsidc.org/DP9/ATLAS/ATL03.004/2018.10.18/ATL03_20181018000228_02960102_004_01.h5\n \n \n \n\n\n\n# We can access the data links with the data_links()\nfor g in granules[0:10]:\n print(g.data_links())" - }, - { - "objectID": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#cloud-collections", - "href": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#cloud-collections", - "title": "Searching and Downloading NSIDC Cloud Collections", - "section": "Cloud Collections", - "text": "Cloud Collections\nSome NSIDC cloud collections are not yet public we need to authenticate ourselves with CMR first.\n\nimport getpass\nimport textwrap\n\nfrom cmr.search import collection as cmr_collection\nfrom cmr.search import granule \nfrom cmr.auth import token\n\nfrom cmr_auth import CMRAuth\n\n# NON_AWS collections are hosted at the NSIDC DAAC data center\n# AWS_CLOUD collections are hosted at AWS S3 us-west-2\nNSIDC_PROVIDERS = {\n 'NSIDC_HOSTED': 'NSIDC_ECS', \n 'AWS_HOSTED':'NSIDC_CPRD'\n}\n\n# Use your own EDL username\nUSER= 'betolink'\n\nprint('Enter your NASA Earthdata login password:')\npassword = getpass.getpass()\nCMR_auth = CMRAuth(USER, password)\n# Token to search private collections on CMR\ncmr_token = CMR_auth.get_token()\n\nEnter your NASA Earthdata login password:\n\n\n ········\n\n\n\n# Now let's start our aunthenticated queries on CMR\nquery = {'short_name':'ATL03',\n 'token': cmr_token,\n 'provider': NSIDC_PROVIDERS['AWS_HOSTED']}\n\ncollections = cmr_collection.search(query)\n\nfor collection in collections[0:3]:\n wrapped_abstract = '\\n'.join(textwrap.wrap(f\"Abstract: {collection['umm']['Abstract']}\", 80)) + '\\n'\n print(f\"concept-id: {collection['meta']['concept-id']}\\n\" +\n f\"Title: {collection['umm']['EntryTitle']}\\n\" +\n wrapped_abstract)\n\nconcept-id: C2027878642-NSIDC_CPRD\nTitle: ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004\nAbstract: This data set (ATL03) contains height above the WGS 84 ellipsoid\n(ITRF2014 reference frame), latitude, longitude, and time for all photons\ndownlinked by the Advanced Topographic Laser Altimeter System (ATLAS) instrument\non board the Ice, Cloud and land Elevation Satellite-2 (ICESat-2) observatory.\nThe ATL03 product was designed to be a single source for all photon data and\nancillary information needed by higher-level ATLAS/ICESat-2 products. As such,\nit also includes spacecraft and instrument parameters and ancillary data not\nexplicitly required for ATL03.\n\n\n\n\n# now that we have the concept-id for our ATL03 in the cloud we do the same thing we did with ATL03 hosted at\nfrom cmr_serializer import QueryResult\n# NSIDC but using the cloud concept-id\n# Jeneau ice sheet\nquery = {'concept-id': 'C2027878642-NSIDC_CPRD',\n 'token': cmr_token,\n 'bounding_box': '-135.1977,58.3325,-133.3410,58.9839'}\n\n# Querying for ATL03 v3 using its concept-id and a bounding box\nresults = granule.search(query, limit=1000)\ngranules = QueryResult(results).items()\n\nprint(f\"Total granules found: {len(results)} \\n\")\n\n# Print the first 3 granules\nfor g in granules[0:3]:\n display(g)\n # You can use: print(g) for the regular text representation.\n\nTotal granules found: 135 \n\n\n\n\n \n Id: ATL03_20181014001049_02350102_004_01.h5\n Collection: {'EntryTitle': 'ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004'}\n Spatial coverage: {'HorizontalSpatialDomain': {'Orbit': {'AscendingCrossing': -127.0482205607256, 'StartLatitude': 27.0, 'StartDirection': 'A', 'EndLatitude': 59.5, 'EndDirection': 'A'}}}\n Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2018-10-14T00:10:49.722Z', 'EndingDateTime': '2018-10-14T00:19:19.918Z'}}\n Size(MB): 1764.5729866027832 \n Data: https://data.nsidc.earthdatacloud.nasa.gov/nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/14/ATL03_20181014001049_02350102_004_01.h5s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/14/ATL03_20181014001049_02350102_004_01.h5\n \n \n \n\n\n\n \n Id: ATL03_20181015124359_02580106_004_01.h5\n Collection: {'EntryTitle': 'ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004'}\n Spatial coverage: {'HorizontalSpatialDomain': {'Orbit': {'AscendingCrossing': 49.70324528818096, 'StartLatitude': 59.5, 'StartDirection': 'D', 'EndLatitude': 27.0, 'EndDirection': 'D'}}}\n Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2018-10-15T12:43:57.696Z', 'EndingDateTime': '2018-10-15T12:52:28.274Z'}}\n Size(MB): 276.2403841018677 \n Data: https://data.nsidc.earthdatacloud.nasa.gov/nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/15/ATL03_20181015124359_02580106_004_01.h5s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/15/ATL03_20181015124359_02580106_004_01.h5\n \n \n \n\n\n\n \n Id: ATL03_20181018000228_02960102_004_01.h5\n Collection: {'EntryTitle': 'ATLAS/ICESat-2 L2A Global Geolocated Photon Data V004'}\n Spatial coverage: {'HorizontalSpatialDomain': {'Orbit': {'AscendingCrossing': -127.82682215638665, 'StartLatitude': 27.0, 'StartDirection': 'A', 'EndLatitude': 59.5, 'EndDirection': 'A'}}}\n Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2018-10-18T00:02:28.717Z', 'EndingDateTime': '2018-10-18T00:10:58.903Z'}}\n Size(MB): 877.0574979782104 \n Data: https://data.nsidc.earthdatacloud.nasa.gov/nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/18/ATL03_20181018000228_02960102_004_01.h5s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/18/ATL03_20181018000228_02960102_004_01.h5\n \n \n \n\n\n\nNOTE: Not all the data granules for NSIDC datasets have been migrated to S3. This might result in different counts between the NSIDC hosted data collections and the ones in AWS S3\n\n# We can list the s3 links but \nfor g in granules[0:10]:\n print(g.data_links(only_s3=True))\n\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/14/ATL03_20181014001049_02350102_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/15/ATL03_20181015124359_02580106_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/18/ATL03_20181018000228_02960102_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/11/05/ATL03_20181105113651_05780106_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/11/07/ATL03_20181107225525_06160102_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/11/09/ATL03_20181109112837_06390106_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/11/11/ATL03_20181111224708_06770102_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/11/15/ATL03_20181115223845_07380102_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/12/04/ATL03_20181204101243_10200106_004_01.h5']\n['s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/12/06/ATL03_20181206213114_10580102_004_01.h5']\n\n\nWe note that our RelatedLinks array now contain links to AWS S3, these are the direct URIs for our data granules in the AWS us-west-2 region." - }, - { - "objectID": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#data-access-using-aws-s3", - "href": "examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html#data-access-using-aws-s3", - "title": "Searching and Downloading NSIDC Cloud Collections", - "section": "Data Access using AWS S3", - "text": "Data Access using AWS S3\n\nIMPORTANT: This section will only work if this notebook is running on the AWS us-west-2 zone\n\nThere is more than one way of accessing data on AWS S3, either downloading it to your local machine using the official client library or using a python library.\nPerformance tip: using the HTTPS URLs will decrease the access performance since these links have to internally be processed by AWS’s content delivery system (CloudFront). To get a better performance we should access the S3:// URLs with BOTO3 or a high level S3 enabled library (i.e. S3FS)\nRelated links: * HDF in the Cloud challenges and solutions for scientific data * Cloud Storage (Amazon S3) HDF5 Connector\n\n# READ only temporary credentials\nimport s3fs\nimport h5py\n\n# This credentials only last 1 hour.\ns3_cred = CMR_auth.get_s3_credentials()\n\n\ns3_fs = s3fs.S3FileSystem(key=s3_cred['accessKeyId'],\n secret=s3_cred['secretAccessKey'],\n token=s3_cred['sessionToken'])\n\n# Now you could grab S3 links to your cloud instance (EC2, Hub etc) using:\n# s3_fs.get('s3://SOME_LOCATION/ATL03_20181015124359_02580106_004_01.h5', 'test.h5')\n\n\nWe now have the propper credentials and file mapper to access the data within AWS us-west-2.\n\nwith s3_fs.open('s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/15/ATL03_20181015124359_02580106_004_01.h5', 'rb') as s3f:\n with h5py.File(s3f, 'r') as f:\n print([key for key in f.keys()])\n\n\n\nUsing xarray to open files on S3\nATL data is complex so xarray doesn’t know how to extract the important bits out of it.\n\nimport xarray\n\nwith s3_fs.open('s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/15/ATL03_20181015124359_02580106_004_01.h5', 'rb') as s3f:\n ds= xarray.open_dataset(s3f)\n for varname in ds:\n print(varname)\nds\n\n\n\n“Downloading” files on S3 using the official aws-cli library\nThe quotes on downloading are because ideally you’ll be working on an EC2 (virtual machine for short) instance on the us-west-2 region." - }, - { - "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html", - "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html", - "title": "GES DISC - MERRA2", - "section": "", - "text": "Many of NASA’s current and legacy data collections are archive in netCDF4 format. By itself, netCDF4 are not cloud optimized and reading these files can take as long from a personal/local work environment as it takes to read the data from a working environment deployed in the cloud. Using Kerchunk, we can treat these files as cloud optimized assets by creating metadata json file describing existing netCDF4 files, their chunks, and where to access them. The json reference files can be read in using Zarr and Xarray for efficient reads and fast processing." - }, - { - "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#requirements", - "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#requirements", - "title": "GES DISC - MERRA2", - "section": "Requirements", - "text": "Requirements\n\n1. AWS instance running in us-west-2\nNASA Earthdata Cloud data in S3 can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region.\n\n\n2. Earthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\n3. netrc File\nYou will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. For additional information see: Authentication for NASA Earthdata." - }, - { - "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#import-required-packages", - "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#import-required-packages", - "title": "GES DISC - MERRA2", - "section": "Import required packages", - "text": "Import required packages\n\nimport requests\nimport xarray as xr\nimport ujson\nimport s3fs\nimport fsspec\nfrom tqdm import tqdm\nfrom glob import glob\nimport os\nimport pathlib\nimport hvplot.xarray\n\nfrom kerchunk.hdf import SingleHdf5ToZarr\nfrom kerchunk.combine import MultiZarrToZarr\n\n# The xarray produced from the reference file throws a SerializationWarning for each variable. Will need to explore why\nimport warnings\nwarnings.simplefilter(\"ignore\")" - }, - { - "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#create-dask-client-to-process-the-output-json-file-in-parallel", - "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#create-dask-client-to-process-the-output-json-file-in-parallel", - "title": "GES DISC - MERRA2", - "section": "Create Dask client to process the output json file in parallel", - "text": "Create Dask client to process the output json file in parallel\nGenerating the Kerchunk reference file can take some time depending on the internal structure of the data. Dask allows us to execute the reference file generation process in parallel, thus speeding up the overall process.\n\nimport dask\nfrom dask.distributed import Client\nclient = Client(n_workers=4)\nclient\n\n\n \n \n Client\n Client-3e6c0be3-d18c-11ec-809e-527eee20f3f0\n \n\n\n\nConnection method: Cluster object\nCluster type: distributed.LocalCluster\n\n\nDashboard: http://127.0.0.1:8787/status\n\n\n\n\n\nCluster Info\n\n\n\n\n\n\nLocalCluster\n64356e89\n\n\n\nDashboard: http://127.0.0.1:8787/status\nWorkers: 4\n\n\nTotal threads: 4\nTotal memory: 7.48 GiB\n\n\nStatus: running\nUsing processes: True\n\n\n\n\n\nScheduler Info\n\n\n\n\n\n\nScheduler\nScheduler-9811e6a9-be9d-4a54-9cc3-f17a8d5ea5bb\n\n\n\nComm: tcp://127.0.0.1:33947\nWorkers: 4\n\n\nDashboard: http://127.0.0.1:8787/status\nTotal threads: 4\n\n\nStarted: Just now\nTotal memory: 7.48 GiB\n\n\n\n\n\n\nWorkers\n\n\n\n\n\nWorker: 0\n\n\n\nComm: tcp://127.0.0.1:38069\nTotal threads: 1\n\n\nDashboard: http://127.0.0.1:44433/status\nMemory: 1.87 GiB\n\n\nNanny: tcp://127.0.0.1:43685\n\n\n\nLocal directory: /home/jovyan/earthdata-cloud-cookbook/examples/GESDISC/dask-worker-space/worker-6b5pkr_f\n\n\n\n\n\n\n\n\n\n\nWorker: 1\n\n\n\nComm: tcp://127.0.0.1:32859\nTotal threads: 1\n\n\nDashboard: http://127.0.0.1:36075/status\nMemory: 1.87 GiB\n\n\nNanny: tcp://127.0.0.1:35291\n\n\n\nLocal directory: /home/jovyan/earthdata-cloud-cookbook/examples/GESDISC/dask-worker-space/worker-piqb4cet\n\n\n\n\n\n\n\n\n\n\nWorker: 2\n\n\n\nComm: tcp://127.0.0.1:36601\nTotal threads: 1\n\n\nDashboard: http://127.0.0.1:45185/status\nMemory: 1.87 GiB\n\n\nNanny: tcp://127.0.0.1:41163\n\n\n\nLocal directory: /home/jovyan/earthdata-cloud-cookbook/examples/GESDISC/dask-worker-space/worker-d_91r19y\n\n\n\n\n\n\n\n\n\n\nWorker: 3\n\n\n\nComm: tcp://127.0.0.1:34309\nTotal threads: 1\n\n\nDashboard: http://127.0.0.1:38301/status\nMemory: 1.87 GiB\n\n\nNanny: tcp://127.0.0.1:44119\n\n\n\nLocal directory: /home/jovyan/earthdata-cloud-cookbook/examples/GESDISC/dask-worker-space/worker-9wa5m1vc" - }, - { - "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#get-temporary-s3-credentials", - "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#get-temporary-s3-credentials", - "title": "GES DISC - MERRA2", - "section": "Get temporary S3 credentials", - "text": "Get temporary S3 credentials\nTemporary S3 credentials need to be passed to AWS. Note, these credentials must be refreshed after 1 hour.\n\ns3_cred_endpoint = {\n 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n 'ornldaac':'https://data.ornldaac.earthdata.nasa.gov/s3credentials',\n 'gesdisc':'https://data.gesdisc.earthdata.nasa.gov/s3credentials'\n}\n\n\ndef get_temp_creds():\n temp_creds_url = s3_cred_endpoint['gesdisc']\n return requests.get(temp_creds_url).json()\n\n\ntemp_creds_req = get_temp_creds()" - }, - { - "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#direct-access-a-single-netcdf4-file", - "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#direct-access-a-single-netcdf4-file", - "title": "GES DISC - MERRA2", - "section": "Direct Access a single netCDF4 file", - "text": "Direct Access a single netCDF4 file\nPass temporary credentials to our filesystem object to access the S3 assets\n\nfs = s3fs.S3FileSystem(\n anon=False,\n key=temp_creds_req['accessKeyId'],\n secret=temp_creds_req['secretAccessKey'],\n token=temp_creds_req['sessionToken']\n)\n\n\nurl = 's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4'\n\n\ns3_file_obj = fs.open(url, mode='rb')\n\nTime how long it takes to directly access a cloud asset for comparisons later.\n\n%%time\n\nxr_ds = xr.open_dataset(s3_file_obj, chunks='auto', engine='h5netcdf')\nxr_ds\n\nCPU times: user 2.9 s, sys: 228 ms, total: 3.13 s\nWall time: 7.53 s\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (lon: 576, lat: 361, time: 24)\nCoordinates:\n * lon (lon) float64 -180.0 -179.4 -178.8 -178.1 ... 178.1 178.8 179.4\n * lat (lat) float64 -90.0 -89.5 -89.0 -88.5 ... 88.5 89.0 89.5 90.0\n * time (time) datetime64[ns] 2019-05-01T00:30:00 ... 2019-05-01T23:30:00\nData variables: (12/47)\n CLDPRS (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n CLDTMP (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n DISPH (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n H1000 (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n H250 (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n H500 (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n ... ...\n V250 (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n V2M (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n V500 (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n V50M (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n V850 (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n ZLCL (time, lat, lon) float32 dask.array<chunksize=(24, 361, 576), meta=np.ndarray>\nAttributes: (12/30)\n History: Original file generated: Sat May 11 22...\n Comment: GMAO filename: d5124_m2_jan10.tavg1_2d...\n Filename: MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4\n Conventions: CF-1\n Institution: NASA Global Modeling and Assimilation ...\n References: http://gmao.gsfc.nasa.gov\n ... ...\n Contact: http://gmao.gsfc.nasa.gov\n identifier_product_doi: 10.5067/VJAFPLI1CSIV\n RangeBeginningDate: 2019-05-01\n RangeBeginningTime: 00:00:00.000000\n RangeEndingDate: 2019-05-01\n RangeEndingTime: 23:59:59.000000xarray.DatasetDimensions:lon: 576lat: 361time: 24Coordinates: (3)lon(lon)float64-180.0 -179.4 ... 178.8 179.4long_name :longitudeunits :degrees_eastvmax :1000000000000000.0vmin :-1000000000000000.0valid_range :[-1.e+15 1.e+15]array([-180. , -179.375, -178.75 , ..., 178.125, 178.75 , 179.375])lat(lat)float64-90.0 -89.5 -89.0 ... 89.5 90.0long_name :latitudeunits :degrees_northvmax :1000000000000000.0vmin :-1000000000000000.0valid_range :[-1.e+15 1.e+15]array([-90. , -89.5, -89. , ..., 89. , 89.5, 90. ])time(time)datetime64[ns]2019-05-01T00:30:00 ... 2019-05-...long_name :timetime_increment :10000begin_date :20190501begin_time :3000vmax :1000000000000000.0vmin :-1000000000000000.0valid_range :[-1.e+15 1.e+15]array(['2019-05-01T00:30:00.000000000', '2019-05-01T01:30:00.000000000',\n '2019-05-01T02:30:00.000000000', '2019-05-01T03:30:00.000000000',\n '2019-05-01T04:30:00.000000000', '2019-05-01T05:30:00.000000000',\n '2019-05-01T06:30:00.000000000', '2019-05-01T07:30:00.000000000',\n '2019-05-01T08:30:00.000000000', '2019-05-01T09:30:00.000000000',\n '2019-05-01T10:30:00.000000000', '2019-05-01T11:30:00.000000000',\n '2019-05-01T12:30:00.000000000', '2019-05-01T13:30:00.000000000',\n '2019-05-01T14:30:00.000000000', '2019-05-01T15:30:00.000000000',\n '2019-05-01T16:30:00.000000000', '2019-05-01T17:30:00.000000000',\n '2019-05-01T18:30:00.000000000', '2019-05-01T19:30:00.000000000',\n '2019-05-01T20:30:00.000000000', '2019-05-01T21:30:00.000000000',\n '2019-05-01T22:30:00.000000000', '2019-05-01T23:30:00.000000000'],\n dtype='datetime64[ns]')Data variables: (47)CLDPRS(time, lat, lon)float32dask.array<chunksize=(24, 361, 576), meta=np.ndarray>long_name :cloud_top_pressureunits :Pafmissing_value :1000000000000000.0standard_name :cloud_top_pressurevmax :1000000000000000.0vmin :-1000000000000000.0valid_range :[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nCLDTMP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ncloud_top_temperature\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ncloud_top_temperature\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nDISPH\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nzero_plane_displacement_height\n\nunits :\n\nm\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nzero_plane_displacement_height\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH1000\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nheight_at_1000_mb\n\nunits :\n\nm\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nheight_at_1000_mb\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nheight_at_250_hPa\n\nunits :\n\nm\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nheight_at_250_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nheight_at_500_hPa\n\nunits :\n\nm\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nheight_at_500_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nheight_at_850_hPa\n\nunits :\n\nm\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nheight_at_850_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nOMEGA500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nomega_at_500_hPa\n\nunits :\n\nPa s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nomega_at_500_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nPBLTOP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\npbltop_pressure\n\nunits :\n\nPa\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\npbltop_pressure\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nPS\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nsurface_pressure\n\nunits :\n\nPa\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nsurface_pressure\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nspecific_humidity_at_250_hPa\n\nunits :\n\nkg kg-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nspecific_humidity_at_250_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nspecific_humidity_at_500_hPa\n\nunits :\n\nkg kg-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nspecific_humidity_at_500_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nspecific_humidity_at_850_hPa\n\nunits :\n\nkg kg-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nspecific_humidity_at_850_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQV10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\n10-meter_specific_humidity\n\nunits :\n\nkg kg-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\n10-meter_specific_humidity\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQV2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\n2-meter_specific_humidity\n\nunits :\n\nkg kg-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\n2-meter_specific_humidity\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSLP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nsea_level_pressure\n\nunits :\n\nPa\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nsea_level_pressure\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\n10-meter_air_temperature\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\n10-meter_air_temperature\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nair_temperature_at_250_hPa\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nair_temperature_at_250_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\n2-meter_air_temperature\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\n2-meter_air_temperature\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2MDEW\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ndew_point_temperature_at_2_m\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ndew_point_temperature_at_2_m\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2MWET\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nwet_bulb_temperature_at_2_m\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nwet_bulb_temperature_at_2_m\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nair_temperature_at_500_hPa\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nair_temperature_at_500_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nair_temperature_at_850_hPa\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nair_temperature_at_850_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTO3\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntotal_column_ozone\n\nunits :\n\nDobsons\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntotal_column_ozone\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTOX\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntotal_column_odd_oxygen\n\nunits :\n\nkg m-2\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntotal_column_odd_oxygen\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQI\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntotal_precipitable_ice_water\n\nunits :\n\nkg m-2\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntotal_precipitable_ice_water\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQL\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntotal_precipitable_liquid_water\n\nunits :\n\nkg m-2\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntotal_precipitable_liquid_water\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQV\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntotal_precipitable_water_vapor\n\nunits :\n\nkg m-2\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntotal_precipitable_water_vapor\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPB\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntropopause_pressure_based_on_blended_estimate\n\nunits :\n\nPa\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntropopause_pressure_based_on_blended_estimate\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPT\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntropopause_pressure_based_on_thermal_estimate\n\nunits :\n\nPa\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntropopause_pressure_based_on_thermal_estimate\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPV\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntropopause_pressure_based_on_EPV_estimate\n\nunits :\n\nPa\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntropopause_pressure_based_on_EPV_estimate\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPQ\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntropopause_specific_humidity_using_blended_TROPP_estimate\n\nunits :\n\nkg kg-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntropopause_specific_humidity_using_blended_TROPP_estimate\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPT\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\ntropopause_temperature_using_blended_TROPP_estimate\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\ntropopause_temperature_using_blended_TROPP_estimate\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTS\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nsurface_skin_temperature\n\nunits :\n\nK\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nsurface_skin_temperature\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\n10-meter_eastward_wind\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\n10-meter_eastward_wind\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\neastward_wind_at_250_hPa\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\neastward_wind_at_250_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\n2-meter_eastward_wind\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\n2-meter_eastward_wind\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\neastward_wind_at_500_hPa\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\neastward_wind_at_500_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU50M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\neastward_wind_at_50_meters\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\neastward_wind_at_50_meters\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\neastward_wind_at_850_hPa\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\neastward_wind_at_850_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\n10-meter_northward_wind\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\n10-meter_northward_wind\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nnorthward_wind_at_250_hPa\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nnorthward_wind_at_250_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\n2-meter_northward_wind\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\n2-meter_northward_wind\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nnorthward_wind_at_500_hPa\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nnorthward_wind_at_500_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV50M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nnorthward_wind_at_50_meters\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nnorthward_wind_at_50_meters\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nnorthward_wind_at_850_hPa\n\nunits :\n\nm s-1\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nnorthward_wind_at_850_hPa\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nZLCL\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(24, 361, 576), meta=np.ndarray>\n\n\n\n\nlong_name :\n\nlifting_condensation_level\n\nunits :\n\nm\n\nfmissing_value :\n\n1000000000000000.0\n\nstandard_name :\n\nlifting_condensation_level\n\nvmax :\n\n1000000000000000.0\n\nvmin :\n\n-1000000000000000.0\n\nvalid_range :\n\n[-1.e+15 1.e+15]\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n19.04 MiB\n\n\nShape\n(24, 361, 576)\n(24, 361, 576)\n\n\nCount\n2 Tasks\n1 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nAttributes: (30)History :Original file generated: Sat May 11 22:08:52 2019 GMTComment :GMAO filename: d5124_m2_jan10.tavg1_2d_slv_Nx.20190501.nc4Filename :MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4Conventions :CF-1Institution :NASA Global Modeling and Assimilation OfficeReferences :http://gmao.gsfc.nasa.govFormat :NetCDF-4/HDF-5SpatialCoverage :globalVersionID :5.12.4TemporalRange :1980-01-01 -> 2016-12-31identifier_product_doi_authority :http://dx.doi.org/ShortName :M2T1NXSLVGranuleID :MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4ProductionDateTime :Original file generated: Sat May 11 22:08:52 2019 GMTLongName :MERRA2 tavg1_2d_slv_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Single-Level DiagnosticsTitle :MERRA2 tavg1_2d_slv_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Single-Level DiagnosticsSouthernmostLatitude :-90.0NorthernmostLatitude :90.0WesternmostLongitude :-180.0EasternmostLongitude :179.375LatitudeResolution :0.5LongitudeResolution :0.625DataResolution :0.5 x 0.625Source :CVS tag: GEOSadas-5_12_4_p16_sp3_M2-OPSContact :http://gmao.gsfc.nasa.govidentifier_product_doi :10.5067/VJAFPLI1CSIVRangeBeginningDate :2019-05-01RangeBeginningTime :00:00:00.000000RangeEndingDate :2019-05-01RangeEndingTime :23:59:59.000000" - }, - { - "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#specify-a-list-of-s3-urls", - "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#specify-a-list-of-s3-urls", - "title": "GES DISC - MERRA2", - "section": "Specify a list of S3 URLs", - "text": "Specify a list of S3 URLs\nData Collection: MERRA2_400.tavg1_2d_slv_Nx\nTime Range: 05/01/2019 - 05/31/2019\n\nurls = ['s3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190502.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190503.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190504.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190505.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190506.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190507.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190508.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190509.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190510.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190511.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190512.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190513.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190514.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190515.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190516.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190517.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190518.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190519.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190520.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190521.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190522.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190523.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190524.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190525.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190526.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190527.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190528.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190529.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190530.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190531.nc4']" - }, - { - "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#generate-the-kerchunk-reference-files.", - "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#generate-the-kerchunk-reference-files.", - "title": "GES DISC - MERRA2", - "section": "Generate the Kerchunk reference files.", - "text": "Generate the Kerchunk reference files.\nDefine a function to generate the Kerchunk reference files. These files can take a little time to generate.\n\ndef gen_json(u):\n so = dict(\n mode= \"rb\", \n anon= False, \n default_fill_cache= False,\n default_cache_type= \"none\"\n )\n with fs.open(u, **so) as infile:\n h5chunks = SingleHdf5ToZarr(infile, u, inline_threshold=300)\n with open(f\"jsons/{u.split('/')[-1]}.json\", 'wb') as outf:\n outf.write(ujson.dumps(h5chunks.translate()).encode())\n\nCreate output jsons directory if one does not exist.\n\npathlib.Path('./jsons/').mkdir(exist_ok=True)\n\nUse the Dask Delayed function to create the Kerchunk reference file for each URL from the list of URLs in parallel\n\n%%time\n\nreference_files = []\nfor url in urls:\n ref = dask.delayed(gen_json)(url)\n reference_files.append(ref)\n\nreference_files_compute = dask.compute(*reference_files)\n\nCPU times: user 29 s, sys: 11.1 s, total: 40 s\nWall time: 11min 6s\n\n\nCreate a python list with the paths to the reference files.\n\nreference_list = sorted(glob('./jsons/*.json'))\nreference_list\n\n['./jsons/MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190502.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190503.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190504.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190505.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190506.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190507.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190508.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190509.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190510.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190511.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190512.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190513.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190514.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190515.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190516.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190517.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190518.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190519.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190520.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190521.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190522.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190523.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190524.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190525.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190526.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190527.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190528.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190529.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190530.nc4.json',\n './jsons/MERRA2_400.tavg1_2d_slv_Nx.20190531.nc4.json']" - }, - { - "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#read-single-netcdf4-using-kerchunk-reference-file", - "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#read-single-netcdf4-using-kerchunk-reference-file", - "title": "GES DISC - MERRA2", - "section": "Read single netCDF4 using Kerchunk reference file", - "text": "Read single netCDF4 using Kerchunk reference file\nOpen the first reference file to read into an xarray dataset\n\nwith open(reference_list[0]) as j:\n reference = ujson.load(j)\n\nSet configurations options\n\ns_opts = {'skip_instance_cache':True} #json\nr_opts = {'anon':False, \n 'key':temp_creds_req['accessKeyId'], \n 'secret':temp_creds_req['secretAccessKey'], \n 'token':temp_creds_req['sessionToken']} #ncfiles\n\n\nfs_single = fsspec.filesystem(\"reference\",\n fo=reference,\n ref_storage_args=s_opts,\n remote_protocol='s3', \n remote_options=r_opts)\n\nRead in a single reference object. We get a lot of SerializationWarnings which are ignored here using the warning package.\nNOTE, the fill value, data range, min value, and max value DO NOT match the source file. Will need to look into this more.\n\n%%time\n\nm = fs_single.get_mapper(\"\")\nds_single = xr.open_dataset(m, engine=\"zarr\", backend_kwargs={'consolidated':False}, chunks={})\nds_single\n\nCPU times: user 142 ms, sys: 3.29 ms, total: 146 ms\nWall time: 354 ms\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (time: 24, lat: 361, lon: 576)\nCoordinates:\n * lat (lat) float64 -90.0 -89.5 -89.0 -88.5 ... 88.5 89.0 89.5 90.0\n * lon (lon) float64 -180.0 -179.4 -178.8 -178.1 ... 178.1 178.8 179.4\n * time (time) datetime64[ns] 2019-05-01T00:30:00 ... 2019-05-01T23:30:00\nData variables: (12/47)\n CLDPRS (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n CLDTMP (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n DISPH (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n H1000 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n H250 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n H500 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n ... ...\n V250 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n V2M (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n V500 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n V50M (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n V850 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n ZLCL (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\nAttributes: (12/30)\n Comment: GMAO filename: d5124_m2_jan10.tavg1_2d...\n Contact: http://gmao.gsfc.nasa.gov\n Conventions: CF-1\n DataResolution: 0.5 x 0.625\n EasternmostLongitude: 179.375\n Filename: MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4\n ... ...\n TemporalRange: 1980-01-01 -> 2016-12-31\n Title: MERRA2 tavg1_2d_slv_Nx: 2d,1-Hourly,Ti...\n VersionID: 5.12.4\n WesternmostLongitude: -180.0\n identifier_product_doi: 10.5067/VJAFPLI1CSIV\n identifier_product_doi_authority: http://dx.doi.org/xarray.DatasetDimensions:time: 24lat: 361lon: 576Coordinates: (3)lat(lat)float64-90.0 -89.5 -89.0 ... 89.5 90.0long_name :latitudeunits :degrees_northvalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array([-90. , -89.5, -89. , ..., 89. , 89.5, 90. ])lon(lon)float64-180.0 -179.4 ... 178.8 179.4long_name :longitudeunits :degrees_eastvalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array([-180. , -179.375, -178.75 , ..., 178.125, 178.75 , 179.375])time(time)datetime64[ns]2019-05-01T00:30:00 ... 2019-05-...begin_date :20190501begin_time :3000long_name :timetime_increment :10000valid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array(['2019-05-01T00:30:00.000000000', '2019-05-01T01:30:00.000000000',\n '2019-05-01T02:30:00.000000000', '2019-05-01T03:30:00.000000000',\n '2019-05-01T04:30:00.000000000', '2019-05-01T05:30:00.000000000',\n '2019-05-01T06:30:00.000000000', '2019-05-01T07:30:00.000000000',\n '2019-05-01T08:30:00.000000000', '2019-05-01T09:30:00.000000000',\n '2019-05-01T10:30:00.000000000', '2019-05-01T11:30:00.000000000',\n '2019-05-01T12:30:00.000000000', '2019-05-01T13:30:00.000000000',\n '2019-05-01T14:30:00.000000000', '2019-05-01T15:30:00.000000000',\n '2019-05-01T16:30:00.000000000', '2019-05-01T17:30:00.000000000',\n '2019-05-01T18:30:00.000000000', '2019-05-01T19:30:00.000000000',\n '2019-05-01T20:30:00.000000000', '2019-05-01T21:30:00.000000000',\n '2019-05-01T22:30:00.000000000', '2019-05-01T23:30:00.000000000'],\n dtype='datetime64[ns]')Data variables: (47)CLDPRS(time, lat, lon)float32dask.array<chunksize=(1, 91, 144), meta=np.ndarray>fmissing_value :999999986991104.0long_name :cloud_top_pressurestandard_name :cloud_top_pressureunits :Pavalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nCLDTMP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ncloud_top_temperature\n\nstandard_name :\n\ncloud_top_temperature\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nDISPH\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nzero_plane_displacement_height\n\nstandard_name :\n\nzero_plane_displacement_height\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH1000\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nheight_at_1000_mb\n\nstandard_name :\n\nheight_at_1000_mb\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nheight_at_250_hPa\n\nstandard_name :\n\nheight_at_250_hPa\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nheight_at_500_hPa\n\nstandard_name :\n\nheight_at_500_hPa\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nheight_at_850_hPa\n\nstandard_name :\n\nheight_at_850_hPa\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nOMEGA500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nomega_at_500_hPa\n\nstandard_name :\n\nomega_at_500_hPa\n\nunits :\n\nPa s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nPBLTOP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\npbltop_pressure\n\nstandard_name :\n\npbltop_pressure\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nPS\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nsurface_pressure\n\nstandard_name :\n\nsurface_pressure\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nspecific_humidity_at_250_hPa\n\nstandard_name :\n\nspecific_humidity_at_250_hPa\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nspecific_humidity_at_500_hPa\n\nstandard_name :\n\nspecific_humidity_at_500_hPa\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nspecific_humidity_at_850_hPa\n\nstandard_name :\n\nspecific_humidity_at_850_hPa\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQV10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n10-meter_specific_humidity\n\nstandard_name :\n\n10-meter_specific_humidity\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQV2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n2-meter_specific_humidity\n\nstandard_name :\n\n2-meter_specific_humidity\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSLP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nsea_level_pressure\n\nstandard_name :\n\nsea_level_pressure\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n10-meter_air_temperature\n\nstandard_name :\n\n10-meter_air_temperature\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nair_temperature_at_250_hPa\n\nstandard_name :\n\nair_temperature_at_250_hPa\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n2-meter_air_temperature\n\nstandard_name :\n\n2-meter_air_temperature\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2MDEW\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ndew_point_temperature_at_2_m\n\nstandard_name :\n\ndew_point_temperature_at_2_m\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2MWET\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nwet_bulb_temperature_at_2_m\n\nstandard_name :\n\nwet_bulb_temperature_at_2_m\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nair_temperature_at_500_hPa\n\nstandard_name :\n\nair_temperature_at_500_hPa\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nair_temperature_at_850_hPa\n\nstandard_name :\n\nair_temperature_at_850_hPa\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTO3\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_column_ozone\n\nstandard_name :\n\ntotal_column_ozone\n\nunits :\n\nDobsons\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTOX\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_column_odd_oxygen\n\nstandard_name :\n\ntotal_column_odd_oxygen\n\nunits :\n\nkg m-2\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQI\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_precipitable_ice_water\n\nstandard_name :\n\ntotal_precipitable_ice_water\n\nunits :\n\nkg m-2\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQL\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_precipitable_liquid_water\n\nstandard_name :\n\ntotal_precipitable_liquid_water\n\nunits :\n\nkg m-2\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQV\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_precipitable_water_vapor\n\nstandard_name :\n\ntotal_precipitable_water_vapor\n\nunits :\n\nkg m-2\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPB\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_pressure_based_on_blended_estimate\n\nstandard_name :\n\ntropopause_pressure_based_on_blended_estimate\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPT\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_pressure_based_on_thermal_estimate\n\nstandard_name :\n\ntropopause_pressure_based_on_thermal_estimate\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPV\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_pressure_based_on_EPV_estimate\n\nstandard_name :\n\ntropopause_pressure_based_on_EPV_estimate\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPQ\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_specific_humidity_using_blended_TROPP_estimate\n\nstandard_name :\n\ntropopause_specific_humidity_using_blended_TROPP_estimate\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPT\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_temperature_using_blended_TROPP_estimate\n\nstandard_name :\n\ntropopause_temperature_using_blended_TROPP_estimate\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTS\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nsurface_skin_temperature\n\nstandard_name :\n\nsurface_skin_temperature\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n10-meter_eastward_wind\n\nstandard_name :\n\n10-meter_eastward_wind\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\neastward_wind_at_250_hPa\n\nstandard_name :\n\neastward_wind_at_250_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n2-meter_eastward_wind\n\nstandard_name :\n\n2-meter_eastward_wind\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\neastward_wind_at_500_hPa\n\nstandard_name :\n\neastward_wind_at_500_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU50M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\neastward_wind_at_50_meters\n\nstandard_name :\n\neastward_wind_at_50_meters\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\neastward_wind_at_850_hPa\n\nstandard_name :\n\neastward_wind_at_850_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n10-meter_northward_wind\n\nstandard_name :\n\n10-meter_northward_wind\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nnorthward_wind_at_250_hPa\n\nstandard_name :\n\nnorthward_wind_at_250_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n2-meter_northward_wind\n\nstandard_name :\n\n2-meter_northward_wind\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nnorthward_wind_at_500_hPa\n\nstandard_name :\n\nnorthward_wind_at_500_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV50M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nnorthward_wind_at_50_meters\n\nstandard_name :\n\nnorthward_wind_at_50_meters\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nnorthward_wind_at_850_hPa\n\nstandard_name :\n\nnorthward_wind_at_850_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nZLCL\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nlifting_condensation_level\n\nstandard_name :\n\nlifting_condensation_level\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n19.04 MiB\n51.19 kiB\n\n\nShape\n(24, 361, 576)\n(1, 91, 144)\n\n\nCount\n385 Tasks\n384 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nAttributes: (30)Comment :GMAO filename: d5124_m2_jan10.tavg1_2d_slv_Nx.20190501.nc4Contact :http://gmao.gsfc.nasa.govConventions :CF-1DataResolution :0.5 x 0.625EasternmostLongitude :179.375Filename :MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4Format :NetCDF-4/HDF-5GranuleID :MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4History :Original file generated: Sat May 11 22:08:52 2019 GMTInstitution :NASA Global Modeling and Assimilation OfficeLatitudeResolution :0.5LongName :MERRA2 tavg1_2d_slv_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Single-Level DiagnosticsLongitudeResolution :0.625NorthernmostLatitude :90.0ProductionDateTime :Original file generated: Sat May 11 22:08:52 2019 GMTRangeBeginningDate :2019-05-01RangeBeginningTime :00:00:00.000000RangeEndingDate :2019-05-01RangeEndingTime :23:59:59.000000References :http://gmao.gsfc.nasa.govShortName :M2T1NXSLVSource :CVS tag: GEOSadas-5_12_4_p16_sp3_M2-OPSSouthernmostLatitude :-90.0SpatialCoverage :globalTemporalRange :1980-01-01 -> 2016-12-31Title :MERRA2 tavg1_2d_slv_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Single-Level DiagnosticsVersionID :5.12.4WesternmostLongitude :-180.0identifier_product_doi :10.5067/VJAFPLI1CSIVidentifier_product_doi_authority :http://dx.doi.org/" - }, - { - "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#read-multiple-netcdf4-files-using-kerchunk-reference-file", - "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#read-multiple-netcdf4-files-using-kerchunk-reference-file", - "title": "GES DISC - MERRA2", - "section": "Read multiple netCDF4 files using Kerchunk reference file", - "text": "Read multiple netCDF4 files using Kerchunk reference file\nCombine the individual reference files into a single time series reference object\n\n%%time\n\nds_k =[]\nfor ref in reference_list:\n s_opts = s_opts\n r_opts = r_opts\n fs = fsspec.filesystem(\"reference\",\n fo=ref,\n ref_storage_args=s_opts,\n remote_protocol='s3',\n remote_options=r_opts)\n m = fs.get_mapper(\"\")\n ds_k.append(xr.open_dataset(m, engine=\"zarr\", backend_kwargs={'consolidated':False}, chunks={}))\n \nds_multi = xr.concat(ds_k, dim='time')\n \nds_multi\n\nCPU times: user 8.93 s, sys: 174 ms, total: 9.1 s\nWall time: 14.9 s\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (time: 744, lat: 361, lon: 576)\nCoordinates:\n * lat (lat) float64 -90.0 -89.5 -89.0 -88.5 ... 88.5 89.0 89.5 90.0\n * lon (lon) float64 -180.0 -179.4 -178.8 -178.1 ... 178.1 178.8 179.4\n * time (time) datetime64[ns] 2019-05-01T00:30:00 ... 2019-05-31T23:30:00\nData variables: (12/47)\n CLDPRS (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n CLDTMP (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n DISPH (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n H1000 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n H250 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n H500 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n ... ...\n V250 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n V2M (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n V500 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n V50M (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n V850 (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n ZLCL (time, lat, lon) float32 dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\nAttributes: (12/30)\n Comment: GMAO filename: d5124_m2_jan10.tavg1_2d...\n Contact: http://gmao.gsfc.nasa.gov\n Conventions: CF-1\n DataResolution: 0.5 x 0.625\n EasternmostLongitude: 179.375\n Filename: MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4\n ... ...\n TemporalRange: 1980-01-01 -> 2016-12-31\n Title: MERRA2 tavg1_2d_slv_Nx: 2d,1-Hourly,Ti...\n VersionID: 5.12.4\n WesternmostLongitude: -180.0\n identifier_product_doi: 10.5067/VJAFPLI1CSIV\n identifier_product_doi_authority: http://dx.doi.org/xarray.DatasetDimensions:time: 744lat: 361lon: 576Coordinates: (3)lat(lat)float64-90.0 -89.5 -89.0 ... 89.5 90.0long_name :latitudeunits :degrees_northvalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array([-90. , -89.5, -89. , ..., 89. , 89.5, 90. ])lon(lon)float64-180.0 -179.4 ... 178.8 179.4long_name :longitudeunits :degrees_eastvalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array([-180. , -179.375, -178.75 , ..., 178.125, 178.75 , 179.375])time(time)datetime64[ns]2019-05-01T00:30:00 ... 2019-05-...begin_date :20190501begin_time :3000long_name :timetime_increment :10000valid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array(['2019-05-01T00:30:00.000000000', '2019-05-01T01:30:00.000000000',\n '2019-05-01T02:30:00.000000000', ..., '2019-05-31T21:30:00.000000000',\n '2019-05-31T22:30:00.000000000', '2019-05-31T23:30:00.000000000'],\n dtype='datetime64[ns]')Data variables: (47)CLDPRS(time, lat, lon)float32dask.array<chunksize=(1, 91, 144), meta=np.ndarray>fmissing_value :999999986991104.0long_name :cloud_top_pressurestandard_name :cloud_top_pressureunits :Pavalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nCLDTMP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ncloud_top_temperature\n\nstandard_name :\n\ncloud_top_temperature\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nDISPH\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nzero_plane_displacement_height\n\nstandard_name :\n\nzero_plane_displacement_height\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH1000\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nheight_at_1000_mb\n\nstandard_name :\n\nheight_at_1000_mb\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nheight_at_250_hPa\n\nstandard_name :\n\nheight_at_250_hPa\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nheight_at_500_hPa\n\nstandard_name :\n\nheight_at_500_hPa\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nH850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nheight_at_850_hPa\n\nstandard_name :\n\nheight_at_850_hPa\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nOMEGA500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nomega_at_500_hPa\n\nstandard_name :\n\nomega_at_500_hPa\n\nunits :\n\nPa s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nPBLTOP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\npbltop_pressure\n\nstandard_name :\n\npbltop_pressure\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nPS\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nsurface_pressure\n\nstandard_name :\n\nsurface_pressure\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nspecific_humidity_at_250_hPa\n\nstandard_name :\n\nspecific_humidity_at_250_hPa\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nspecific_humidity_at_500_hPa\n\nstandard_name :\n\nspecific_humidity_at_500_hPa\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQ850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nspecific_humidity_at_850_hPa\n\nstandard_name :\n\nspecific_humidity_at_850_hPa\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQV10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n10-meter_specific_humidity\n\nstandard_name :\n\n10-meter_specific_humidity\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nQV2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n2-meter_specific_humidity\n\nstandard_name :\n\n2-meter_specific_humidity\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nSLP\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nsea_level_pressure\n\nstandard_name :\n\nsea_level_pressure\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n10-meter_air_temperature\n\nstandard_name :\n\n10-meter_air_temperature\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nair_temperature_at_250_hPa\n\nstandard_name :\n\nair_temperature_at_250_hPa\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n2-meter_air_temperature\n\nstandard_name :\n\n2-meter_air_temperature\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2MDEW\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ndew_point_temperature_at_2_m\n\nstandard_name :\n\ndew_point_temperature_at_2_m\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT2MWET\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nwet_bulb_temperature_at_2_m\n\nstandard_name :\n\nwet_bulb_temperature_at_2_m\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nair_temperature_at_500_hPa\n\nstandard_name :\n\nair_temperature_at_500_hPa\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nT850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nair_temperature_at_850_hPa\n\nstandard_name :\n\nair_temperature_at_850_hPa\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTO3\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_column_ozone\n\nstandard_name :\n\ntotal_column_ozone\n\nunits :\n\nDobsons\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTOX\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_column_odd_oxygen\n\nstandard_name :\n\ntotal_column_odd_oxygen\n\nunits :\n\nkg m-2\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQI\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_precipitable_ice_water\n\nstandard_name :\n\ntotal_precipitable_ice_water\n\nunits :\n\nkg m-2\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQL\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_precipitable_liquid_water\n\nstandard_name :\n\ntotal_precipitable_liquid_water\n\nunits :\n\nkg m-2\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTQV\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntotal_precipitable_water_vapor\n\nstandard_name :\n\ntotal_precipitable_water_vapor\n\nunits :\n\nkg m-2\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPB\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_pressure_based_on_blended_estimate\n\nstandard_name :\n\ntropopause_pressure_based_on_blended_estimate\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPT\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_pressure_based_on_thermal_estimate\n\nstandard_name :\n\ntropopause_pressure_based_on_thermal_estimate\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPPV\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_pressure_based_on_EPV_estimate\n\nstandard_name :\n\ntropopause_pressure_based_on_EPV_estimate\n\nunits :\n\nPa\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPQ\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_specific_humidity_using_blended_TROPP_estimate\n\nstandard_name :\n\ntropopause_specific_humidity_using_blended_TROPP_estimate\n\nunits :\n\nkg kg-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTROPT\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\ntropopause_temperature_using_blended_TROPP_estimate\n\nstandard_name :\n\ntropopause_temperature_using_blended_TROPP_estimate\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nTS\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nsurface_skin_temperature\n\nstandard_name :\n\nsurface_skin_temperature\n\nunits :\n\nK\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n10-meter_eastward_wind\n\nstandard_name :\n\n10-meter_eastward_wind\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\neastward_wind_at_250_hPa\n\nstandard_name :\n\neastward_wind_at_250_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n2-meter_eastward_wind\n\nstandard_name :\n\n2-meter_eastward_wind\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\neastward_wind_at_500_hPa\n\nstandard_name :\n\neastward_wind_at_500_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU50M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\neastward_wind_at_50_meters\n\nstandard_name :\n\neastward_wind_at_50_meters\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nU850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\neastward_wind_at_850_hPa\n\nstandard_name :\n\neastward_wind_at_850_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV10M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n10-meter_northward_wind\n\nstandard_name :\n\n10-meter_northward_wind\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV250\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nnorthward_wind_at_250_hPa\n\nstandard_name :\n\nnorthward_wind_at_250_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV2M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\n2-meter_northward_wind\n\nstandard_name :\n\n2-meter_northward_wind\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV500\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nnorthward_wind_at_500_hPa\n\nstandard_name :\n\nnorthward_wind_at_500_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV50M\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nnorthward_wind_at_50_meters\n\nstandard_name :\n\nnorthward_wind_at_50_meters\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nV850\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nnorthward_wind_at_850_hPa\n\nstandard_name :\n\nnorthward_wind_at_850_hPa\n\nunits :\n\nm s-1\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\n\nZLCL\n\n\n(time, lat, lon)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\nfmissing_value :\n\n999999986991104.0\n\nlong_name :\n\nlifting_condensation_level\n\nstandard_name :\n\nlifting_condensation_level\n\nunits :\n\nm\n\nvalid_range :\n\n[-999999986991104.0, 999999986991104.0]\n\nvmax :\n\n999999986991104.0\n\nvmin :\n\n-999999986991104.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\n\nAttributes: (30)Comment :GMAO filename: d5124_m2_jan10.tavg1_2d_slv_Nx.20190501.nc4Contact :http://gmao.gsfc.nasa.govConventions :CF-1DataResolution :0.5 x 0.625EasternmostLongitude :179.375Filename :MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4Format :NetCDF-4/HDF-5GranuleID :MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4History :Original file generated: Sat May 11 22:08:52 2019 GMTInstitution :NASA Global Modeling and Assimilation OfficeLatitudeResolution :0.5LongName :MERRA2 tavg1_2d_slv_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Single-Level DiagnosticsLongitudeResolution :0.625NorthernmostLatitude :90.0ProductionDateTime :Original file generated: Sat May 11 22:08:52 2019 GMTRangeBeginningDate :2019-05-01RangeBeginningTime :00:00:00.000000RangeEndingDate :2019-05-01RangeEndingTime :23:59:59.000000References :http://gmao.gsfc.nasa.govShortName :M2T1NXSLVSource :CVS tag: GEOSadas-5_12_4_p16_sp3_M2-OPSSouthernmostLatitude :-90.0SpatialCoverage :globalTemporalRange :1980-01-01 -> 2016-12-31Title :MERRA2 tavg1_2d_slv_Nx: 2d,1-Hourly,Time-Averaged,Single-Level,Assimilation,Single-Level DiagnosticsVersionID :5.12.4WesternmostLongitude :-180.0identifier_product_doi :10.5067/VJAFPLI1CSIVidentifier_product_doi_authority :http://dx.doi.org/\n\n\nAgains, the fill value, data range, min value, and max value DO NOT match the source file. TODO: explore why the values are different\n\nds_multi['T500']\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray 'T500' (time: 744, lat: 361, lon: 576)>\ndask.array<concatenate, shape=(744, 361, 576), dtype=float32, chunksize=(1, 91, 144), chunktype=numpy.ndarray>\nCoordinates:\n * lat (lat) float64 -90.0 -89.5 -89.0 -88.5 -88.0 ... 88.5 89.0 89.5 90.0\n * lon (lon) float64 -180.0 -179.4 -178.8 -178.1 ... 178.1 178.8 179.4\n * time (time) datetime64[ns] 2019-05-01T00:30:00 ... 2019-05-31T23:30:00\nAttributes:\n fmissing_value: 999999986991104.0\n long_name: air_temperature_at_500_hPa\n standard_name: air_temperature_at_500_hPa\n units: K\n valid_range: [-999999986991104.0, 999999986991104.0]\n vmax: 999999986991104.0\n vmin: -999999986991104.0xarray.DataArray'T500'time: 744lat: 361lon: 576dask.array<chunksize=(1, 91, 144), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n590.15 MiB\n51.19 kiB\n\n\nShape\n(744, 361, 576)\n(1, 91, 144)\n\n\nCount\n23839 Tasks\n11904 Chunks\n\n\nType\nfloat32\nnumpy.ndarray\n\n\n\n\n\n\n\nCoordinates: (3)lat(lat)float64-90.0 -89.5 -89.0 ... 89.5 90.0long_name :latitudeunits :degrees_northvalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array([-90. , -89.5, -89. , ..., 89. , 89.5, 90. ])lon(lon)float64-180.0 -179.4 ... 178.8 179.4long_name :longitudeunits :degrees_eastvalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array([-180. , -179.375, -178.75 , ..., 178.125, 178.75 , 179.375])time(time)datetime64[ns]2019-05-01T00:30:00 ... 2019-05-...begin_date :20190501begin_time :3000long_name :timetime_increment :10000valid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0array(['2019-05-01T00:30:00.000000000', '2019-05-01T01:30:00.000000000',\n '2019-05-01T02:30:00.000000000', ..., '2019-05-31T21:30:00.000000000',\n '2019-05-31T22:30:00.000000000', '2019-05-31T23:30:00.000000000'],\n dtype='datetime64[ns]')Attributes: (7)fmissing_value :999999986991104.0long_name :air_temperature_at_500_hPastandard_name :air_temperature_at_500_hPaunits :Kvalid_range :[-999999986991104.0, 999999986991104.0]vmax :999999986991104.0vmin :-999999986991104.0\n\n\n\n# Commenting for quarto site render\n# ds_multi['T500'].hvplot.image(x='lon', y='lat')" - }, - { - "objectID": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#references", - "href": "examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html#references", - "title": "GES DISC - MERRA2", - "section": "References", - "text": "References\n\nhttps://github.com/fsspec/kerchunk\nhttps://medium.com/pangeo/fake-it-until-you-make-it-reading-goes-netcdf4-data-on-aws-s3-as-zarr-for-rapid-data-access-61e33f8fe685\nhttps://medium.com/pangeo/cloud-performant-reading-of-netcdf4-hdf5-data-using-the-zarr-library-1a95c5c92314" - }, - { - "objectID": "examples/LPDAAC/LPDAAC_ECOSTRESS_LSTE__Kerchunk.html", - "href": "examples/LPDAAC/LPDAAC_ECOSTRESS_LSTE__Kerchunk.html", - "title": "LP DAAC ECOSTRESS", - "section": "", - "text": "!mamba install kerchunk -y --quiet\n\n\n!mamba install h5py=3.2 -y --quiet # Default version in this environment does not work. Must update\n\n\nimport numpy as np\nimport xarray as xr\nimport matplotlib.pyplot as plt\nimport s3fs\nimport datetime as dt\nimport logging\nimport fsspec\nimport ujson\nimport requests\nfrom tqdm import tqdm\nfrom glob import glob\nimport os\nimport hvplot.xarray\n\n# The xarray produced from the reference file throws a SerializationWarning for each variable. Will need to explore why\n#import warnings\n#warnings.simplefilter(\"ignore\")\n\n\n\n\n\n\n\n\n\n\n\n\nfrom kerchunk.hdf import SingleHdf5ToZarr\nfrom kerchunk.combine import MultiZarrToZarr\n\n\ns3_cred_endpoint = {\n 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n 'ornldaac':'https://data.ornldaac.earthdata.nasa.gov/s3credentials',\n 'gesdisc':'https://data.gesdisc.earthdata.nasa.gov/s3credentials'\n}\n\n\ndef get_temp_creds():\n temp_creds_url = s3_cred_endpoint['lpdaac']\n return requests.get(temp_creds_url).json()\n\n\ntemp_creds_req = get_temp_creds()\n\nDirect Access a Single file\n\nfs = s3fs.S3FileSystem(\n anon=False,\n key=temp_creds_req['accessKeyId'],\n secret=temp_creds_req['secretAccessKey'],\n token=temp_creds_req['sessionToken']\n)\n\n\nurl = 's3://lp-prod-protected/ECO_L1B_GEO.002/ECOv002_L1B_GEO_21547_021_20220424T215449_0700_01/ECOv002_L1B_GEO_21547_021_20220424T215449_0700_01.h5'\n\n\ns3_file_obj = fs.open(url, mode='rb')\n\n\nimport h5py\n\n\nh5_file = h5py.File(s3_file_obj)\n\n\n#list(h5_file['Geolocation']['latitude'])\n\n\n%%time\nxr_ds = xr.open_dataset(s3_file_obj, chunks='auto', engine='h5netcdf', backend_kwargs={\"mask_and_scale\" : False, \"decode_times\" : False, \"decode_timedelta\" : False, \"use_cftime\" : False, \"decode_coords\" : False})\nxr_ds\n\nCPU times: user 134 ms, sys: 35 ms, total: 169 ms\nWall time: 534 ms\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: ()\nData variables:\n *empty*xarray.DatasetDimensions:Coordinates: (0)Data variables: (0)Attributes: (0)\n\n\n\nSpecify a list of S3 URLs (MERRA2, 05/01/2019 - 05/31/2019)\n\nurls = ['s3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190502.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190503.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190504.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190505.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190506.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190507.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190508.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190509.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190510.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190511.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190512.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190513.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190514.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190515.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190516.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190517.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190518.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190519.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190520.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190521.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190522.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190523.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190524.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190525.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190526.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190527.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190528.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190529.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190530.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190531.nc4']\n\nGenerate the reference kerchunk files. These files can take a little time to generate. Do not run if JSON files already exist in ./jsons/ directory.\n\nso = dict(\n mode=\"rb\",\n anon=False,\n default_fill_cache=False,\n default_cache_type=\"none\"\n)\n\n#references = []\nfor u in urls:\n with fs.open(u, **so) as infile:\n h5chunks = SingleHdf5ToZarr(infile, u, inline_threshold=300)\n with open(f\"jsons/{u.split('/')[-1]}.json\", 'wb') as outfile:\n translate = h5chunks.translate()\n outfile.write(ujson.dumps(translate).encode())\n #references.append(translate)\n\nCreate a list with the paths to the reference files\n\nfs_ref_list = fsspec.filesystem('file')\n\n\nreference_list = sorted([x for x in fs_ref_list.ls('jsons') if '.json' in x])\nreference_list\n\nOpen the first reference file to read into an xarray dataset\n\nwith open(reference_list[0]) as j:\n reference = ujson.load(j)\n\nSet configurations options\n\ns_opts = {'skip_instance_cache':True} #json\nr_opts = {'anon':False, \n 'key':temp_creds_req['accessKeyId'], \n 'secret':temp_creds_req['secretAccessKey'], \n 'token':temp_creds_req['sessionToken']} #ncfiles\n\n\nfs_single = fsspec.filesystem(\"reference\", \n fo=reference, \n ref_storage_args=s_opts,\n remote_protocol='s3', \n remote_options=r_opts)\n\nRead in a single reference object. Get lots of SerializationWarnings. Ignored using the warning package, but the fill value, data range, min value, and max value DO NOT match the source file.\n\n%%time\nm = fs_single.get_mapper(\"\")\nds_single = xr.open_dataset(m, engine=\"zarr\", backend_kwargs={'consolidated':False}, chunks={})\nds_single\n\n\n#ds_single\n\n\nCombine the single reference files created above into a single time series reference file THIS DID NOT WORK\n\n# mzz = MultiZarrToZarr(\n# reference_list,\n# remote_protocol=\"s3\",\n# remote_options=r_opts,\n# concat_dims=[\"time\"]\n# )\n\n# out = mzz.translate()\n\n\n# fs_mzz = fsspec.filesystem(\"reference\",\n# fo=out,\n# ref_storage_args=s_opts,\n# remote_protocol='s3',\n# remote_options=r_opts)\n\n\n# m = fs_mzz.get_mapper(\"\")\n# ds_multi = xr.open_dataset(m, engine=\"zarr\", backend_kwargs={'consolidated':False}, chunks={})\n\n\nCombine the single reference files created above into a single time series reference file THIS DID WORK\n\n%%time\nds_k =[]\n\nfor ref in reference_list:\n s_opts = s_opts\n r_opts = r_opts\n fs = fsspec.filesystem(\n \"reference\", \n fo=ref, \n ref_storage_args=s_opts,\n remote_protocol='s3', \n remote_options=r_opts)\n m = fs.get_mapper(\"\")\n ds_k.append(xr.open_dataset(m, engine=\"zarr\", backend_kwargs={'consolidated':False}, chunks={}))\n \nds_multi = xr.concat(ds_k, dim='time')\n \nds_multi\n\nAgains, the fill value, data range, min value, and max value DO NOT match the source file\n\n#ds_multi = xr.concat(ds_k, dim='time')\n\n\n#ds_multi\n\n\nds_multi['T500']\n\n\n# Commenting for quarto site render\n# ds_multi['T500'].hvplot.image(x='lon', y='lat')" - }, - { - "objectID": "our-cookbook.html", - "href": "our-cookbook.html", - "title": "Our Cookbook", - "section": "", - "text": "Our Cookbook is a place to learn, share, and experiment with NASA Earthdata on the Cloud. We know this has a lot of moving parts, and we are iterating as we go, and welcome feedback and contributions.\nThe Cookbook has How To building blocks and Tutorials that connect these building blocks through an example research question and specific data. How To guides and Tutorials are stable approaches we’ve developed and used to teach; they have been iterated and improved through feedback from researchers during events we’ve led. We also share work In Development: primarily lessons and other works-in-progress that we’re developing.\nWorking with NASA Earthdata in the Cloud means using a combination of software, tools, many of which require coding and are unfamiliar when we first get started. This is true for us all; we’re all coming with different skills and backgrounds and you’re not alone as we all learn these new technologies and workflows. We have found it helpful to have a growth mindset - these approaches are new and we don’t know how to do them yet. Please, don’t struggle alone - know that we’re all in this together as part of the open source community learning and co-creating together as we migrate our research to the Cloud.\nYou’re able to reuse any code in this book, adapting it for your own purposes. All chapters in the book are a combination of narrative, links, code, and outputs — and underlying each chapter is a file type that you can access on NASA-Openscapes GitHub (linked on the left navbar and right margin of each page): Markdown (.md), Jupyter (.ipynb), RMarkdown (.rmd) or Quarto (.qmd) files.\nWe will also be adding citations throughout so that you can easily credit the NASA Openscapes Mentors who created them." - }, - { - "objectID": "our-cookbook.html#how-to-use-this-earthdata-cloud-cookbook", - "href": "our-cookbook.html#how-to-use-this-earthdata-cloud-cookbook", - "title": "Our Cookbook", - "section": "", - "text": "Our Cookbook is a place to learn, share, and experiment with NASA Earthdata on the Cloud. We know this has a lot of moving parts, and we are iterating as we go, and welcome feedback and contributions.\nThe Cookbook has How To building blocks and Tutorials that connect these building blocks through an example research question and specific data. How To guides and Tutorials are stable approaches we’ve developed and used to teach; they have been iterated and improved through feedback from researchers during events we’ve led. We also share work In Development: primarily lessons and other works-in-progress that we’re developing.\nWorking with NASA Earthdata in the Cloud means using a combination of software, tools, many of which require coding and are unfamiliar when we first get started. This is true for us all; we’re all coming with different skills and backgrounds and you’re not alone as we all learn these new technologies and workflows. We have found it helpful to have a growth mindset - these approaches are new and we don’t know how to do them yet. Please, don’t struggle alone - know that we’re all in this together as part of the open source community learning and co-creating together as we migrate our research to the Cloud.\nYou’re able to reuse any code in this book, adapting it for your own purposes. All chapters in the book are a combination of narrative, links, code, and outputs — and underlying each chapter is a file type that you can access on NASA-Openscapes GitHub (linked on the left navbar and right margin of each page): Markdown (.md), Jupyter (.ipynb), RMarkdown (.rmd) or Quarto (.qmd) files.\nWe will also be adding citations throughout so that you can easily credit the NASA Openscapes Mentors who created them." - }, - { - "objectID": "our-cookbook.html#get-started", - "href": "our-cookbook.html#get-started", - "title": "Our Cookbook", - "section": "Get Started", - "text": "Get Started\nSo, you want to get started working with NASA Earthdata in the cloud? You’ve come to the right place. Here you’ll find resources that can be considered precursors to the how to’s, tutorials, and other guidance you will find across our Cookbook.\n\nEarthdata Login\nTo access NASA Earthdata, whether through your web browser or the cloud, you must first register for an Earthdata Login (EDL) user profile. Once registered, you can use your login credentials to get data through multiple access points. Read about EDL and get registered by following the directions on the Welcome to Earthdata Login page.\n\n\nCoding Essentials\nTo access the cloud programmatically we must have a basic understanding of how to code using common, cloud-relevant languages. Most scientists who work with Earth data use either Python or R already, so we focus on those languages. Python and R are both open-source programming languages widely used in web applications, software development, data science, and machine learning. They are popular because they are free, efficient, have online resources to learn, and can run on most platforms. If you are new to coding, we recommend you participate in a Carpentries Workshop or use open-source resources to teach yourself.\n\n\nBash & Git\nCloud services often are connected to and operated through Bash, a command-line interface and language you’ll see called the terminal, the command line, or the shell. Git is a commonly used version control system that is accessible through Bash. Version control is important for data collaboration because it allows changes by multiple people to be tracked and merged into one source.\n\nThe Unix Shell These lessons will introduce you the shell, a fundamental tool for performing a wide range of computing tasks.\nVersion Control with Git This tutorial will introduce you to Git, a popular open source distributed version control system.\n\n\n\nPython\n\nThe Python Tutorial This tutorial introduces the reader informally to the basic concepts and features of the Python language and system.\nPythia Foundations This book is intended to educate the reader on the essentials for using the Scientific Python Ecosystem (SPE): a collection of open source Python packages that support analysis, manipulation, and visualization of scientific data.\n\n\n\nR\n\nR for Excel Users This course is a friendly intro to becoming a modern R user, full of tidyverse, RMarkdown, GitHub, collaboration, and reproducibility.\nR for Data Science This book will teach you how to do data science with R, including how to get your data into R, get it into the most useful structure, transform it, and visualize.\n\n\n\nCloud-Native Geospatial Formats Guide\nIf you are wondering “why cloud?” and / or wish to learn more about cloud-native geospatial formats, please visit https://guide.cloudnativegeo.org/.\nFor advanced coding guidance, see our How To’s, Tutorials, and Appendix." - }, - { - "objectID": "examples/USGS_Landsat/Landsat_C2_Search_Access.html", - "href": "examples/USGS_Landsat/Landsat_C2_Search_Access.html", - "title": "USGS Landsat Cloud Data - Search and Access", - "section": "", - "text": "import rasterio as rio\nfrom rasterio.session import AWSSession\nimport rioxarray\nimport boto3\nimport json\nfrom pystac_client import Client\nimport hvplot.xarray\nSeach for LS Scenes using STAC API\nstac_url = 'https://landsatlook.usgs.gov/stac-server'\nls_cat = Client.open(stac_url)\nls_cat\n\n<Client id=stac-server>\n[f'{x.title} --- {x.id}' for x in ls_cat.get_children()]\n\n['Landsat Collection 1 Level-2 Albers Surface Temperature (ST) Product --- landsat-c1l2alb-st',\n 'Landsat Collection 2 Level-2 UTM Surface Reflectance (SR) Product --- landsat-c2l2-sr',\n 'Landsat Collection 2 Level-2 UTM Surface Temperature (ST) Product --- landsat-c2l2-st',\n 'Landsat Collection 2 Analysis Ready Data (ARD) Level-2 UTM Surface Temperature (ST) Product --- landsat-c2ard-st',\n 'Landsat Collection 2 Level-2 Albers Top of Atmosphere Brightness Temperature (BT) Product --- landsat-c2l2alb-bt',\n 'Landsat Collection 2 Analysis Ready Data (ARD) Level-2 UTM Top of Atmosphere Brightness Temperature (BT) Product --- landsat-c2ard-bt',\n 'Landsat Collection 2 Level-3 Fractional Snow Covered Area (fSCA) Product --- landsat-c2l3-fsca',\n 'Landsat Collection 1 Level-2 Albers Top of Atmosphere Brightness Temperature (BT) Product --- landsat-c1l2alb-bt',\n 'Landsat Collection 2 Level-1 Product --- landsat-c2l1',\n 'Landsat Collection 2 Level-3 Burned Area (BA) Product --- landsat-c2l3-ba',\n 'Landsat Collection 1 Level-2 Albers Top of Atmosphere (TA) Reflectance Product --- landsat-c1l2alb-ta',\n 'Landsat Collection 2 Level-2 Albers Surface Temperature (ST) Product --- landsat-c2l2alb-st',\n 'Landsat Collection 2 Analysis Ready Data (ARD) Level-2 UTM Surface Reflectance (SR) Product --- landsat-c2ard-sr',\n 'Landsat Collection 1 Level-1 Product --- landsat-c1l1',\n 'Landsat Collection 1 Level-2 Albers Surface Reflectance (SR) Product --- landsat-c1l2alb-sr',\n 'Landsat Collection 2 Level-2 Albers Top of Atmosphere (TA) Reflectance Product --- landsat-c2l2alb-ta',\n 'Landsat Collection 2 Level-2 Albers Surface Reflectance (SR) Product --- landsat-c2l2alb-sr',\n 'Landsat Collection 2 Analysis Ready Data (ARD) Level-2 UTM Top of Atmosphere (TA) Reflectance Product --- landsat-c2ard-ta',\n 'Landsat Collection 2 Level-3 Dynamic Surface Water Extent (DSWE) Product --- landsat-c2l3-dswe']\nlist(ls_cat.get_all_collections())\n\n[<CollectionClient id=landsat-c1l2alb-st>,\n <CollectionClient id=landsat-c2l2-sr>,\n <CollectionClient id=landsat-c2l2-st>,\n <CollectionClient id=landsat-c2ard-st>,\n <CollectionClient id=landsat-c2l2alb-bt>,\n <CollectionClient id=landsat-c2ard-bt>,\n <CollectionClient id=landsat-c2l3-fsca>,\n <CollectionClient id=landsat-c1l2alb-bt>,\n <CollectionClient id=landsat-c2l1>,\n <CollectionClient id=landsat-c2l3-ba>,\n <CollectionClient id=landsat-c1l2alb-ta>,\n <CollectionClient id=landsat-c2l2alb-st>,\n <CollectionClient id=landsat-c2ard-sr>,\n <CollectionClient id=landsat-c1l1>,\n <CollectionClient id=landsat-c1l2alb-sr>,\n <CollectionClient id=landsat-c2l2alb-ta>,\n <CollectionClient id=landsat-c2l2alb-sr>,\n <CollectionClient id=landsat-c2ard-ta>,\n <CollectionClient id=landsat-c2l3-dswe>]\nbbox = [-81.96624,26.38141,-81.80351,26.56457]\ndatetime_range = '2021-10-17T00:00:00Z/2021-10-31T23:59:59Z'\ndatetime_range\n\n'2021-10-17T00:00:00Z/2021-10-31T23:59:59Z'\nsearch = ls_cat.search(\n collections = ['landsat-c2l1'], # Landsat C2 L1 - Copied ID from above. Must be in a list\n bbox = bbox, # Specified above\n datetime = datetime_range # Specified above\n)\nsearch.matched()\n\n3\nitems = list(search.get_items())\nPrint first 5 item ids and associated datetime\n[f'{x.id} --- {x.datetime}' for x in items][:5]\n\n['LC08_L1TP_016042_20211027_20211104_02_T1 --- 2021-10-27 15:56:38.874611+00:00',\n 'LC08_L1TP_016041_20211027_20211104_02_T1 --- 2021-10-27 15:56:14.983570+00:00',\n 'LE07_L1TP_016042_20211019_20211114_02_T1 --- 2021-10-19 14:50:36.703778+00:00']\nhttp_links = []\n\nfor i in items:\n for a in i.assets.values():\n a_link = json.loads(json.dumps(a.to_dict()))\n if a_link['href'].endswith('.TIF'):\n http_links.append(a_link['href'])\nlen(http_links)\n\n49\nhttp_links[:10]\n\n['https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B1.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B2.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B3.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B4.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B5.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B6.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B7.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B8.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B9.TIF',\n 'https://landsatlook.usgs.gov/data/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B10.TIF']\ns3_links = []\n\nfor i in items:\n for a in i.assets.values():\n a_link = json.loads(json.dumps(a.to_dict()))\n if 'alternate' in a_link:\n s3_asset = a_link['alternate']['s3']['href']\n if s3_asset.endswith('.TIF'):\n s3_links.append(s3_asset)\nlen(s3_links)\n\n49\ns3_links[:10]\n\n['s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B1.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B2.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B3.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B4.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B5.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B6.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B7.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B8.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B9.TIF',\n 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B10.TIF']" - }, - { - "objectID": "examples/USGS_Landsat/Landsat_C2_Search_Access.html#data-access---rioxarray", - "href": "examples/USGS_Landsat/Landsat_C2_Search_Access.html#data-access---rioxarray", - "title": "USGS Landsat Cloud Data - Search and Access", - "section": "Data Access - rioxarray", - "text": "Data Access - rioxarray\nSet working environment\n\nsession = boto3.Session()\n\n\nrio_env = rio.Env(AWSSession(session, requester_pays=True), \n AWS_NO_SIGN_REQUEST='NO',\n GDAL_DISABLE_READDIR_ON_OPEN='TRUE')\nrio_env.__enter__()\n\n<rasterio.env.Env at 0x7f6e5276eb50>\n\n\n\ns3_url = 's3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B1.TIF'\n\nRead S3 asset as xarray datarray\n\nda = rioxarray.open_rasterio(s3_url, chunks='auto').squeeze('band', drop=True)\n\n\nda\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (y: 7821, x: 7661)>\ndask.array<getitem, shape=(7821, 7661), dtype=uint16, chunksize=(7821, 7661), chunktype=numpy.ndarray>\nCoordinates:\n * x (x) float64 2.889e+05 2.889e+05 ... 5.187e+05 5.187e+05\n * y (y) float64 2.992e+06 2.992e+06 ... 2.758e+06 2.758e+06\n spatial_ref int64 0\nAttributes:\n _FillValue: 0.0\n scale_factor: 1.0\n add_offset: 0.0xarray.DataArrayy: 7821x: 7661dask.array<chunksize=(7821, 7661), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n114.28 MiB\n114.28 MiB\n\n\nShape\n(7821, 7661)\n(7821, 7661)\n\n\nCount\n3 Tasks\n1 Chunks\n\n\nType\nuint16\nnumpy.ndarray\n\n\n\n\n\n\n\nCoordinates: (3)x(x)float642.889e+05 2.889e+05 ... 5.187e+05array([288900., 288930., 288960., ..., 518640., 518670., 518700.])y(y)float642.992e+06 2.992e+06 ... 2.758e+06array([2992200., 2992170., 2992140., ..., 2757660., 2757630., 2757600.])spatial_ref()int640crs_wkt :PROJCS[\"WGS 84 / UTM zone 17N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-81],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32617\"]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :WGS 84horizontal_datum_name :World Geodetic System 1984projected_crs_name :WGS 84 / UTM zone 17Ngrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-81.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"WGS 84 / UTM zone 17N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-81],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32617\"]]GeoTransform :288885.0 30.0 0.0 2992215.0 0.0 -30.0array(0)Attributes: (3)_FillValue :0.0scale_factor :1.0add_offset :0.0\n\n\nPlot scene\n\n#da.hvplot(x='x', y='y', crs='epsg:32617', rasterize=True, width=800, height=600, tiles = 'ESRI') # TODO: how to exclude fill values\nda.hvplot(x='x', y='y', crs='epsg:32617', rasterize=True, width=800, height=600)" - }, - { - "objectID": "examples/USGS_Landsat/Landsat_C2_Search_Access.html#aws-cli", - "href": "examples/USGS_Landsat/Landsat_C2_Search_Access.html#aws-cli", - "title": "USGS Landsat Cloud Data - Search and Access", - "section": "AWS CLI", - "text": "AWS CLI\nList content in bucket\n\n! aws s3 ls s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/ --request-payer requester\n\n2021-11-05 08:37:33 116979 LC08_L1TP_016042_20211027_20211104_02_T1_ANG.txt\n2021-11-05 08:37:33 77713726 LC08_L1TP_016042_20211027_20211104_02_T1_B1.TIF\n2021-11-05 08:37:34 76108899 LC08_L1TP_016042_20211027_20211104_02_T1_B10.TIF\n2021-11-05 08:37:36 75282069 LC08_L1TP_016042_20211027_20211104_02_T1_B11.TIF\n2021-11-05 08:37:37 77374352 LC08_L1TP_016042_20211027_20211104_02_T1_B2.TIF\n2021-11-05 08:37:38 77527685 LC08_L1TP_016042_20211027_20211104_02_T1_B3.TIF\n2021-11-05 08:37:39 76485245 LC08_L1TP_016042_20211027_20211104_02_T1_B4.TIF\n2021-11-05 08:37:40 78304185 LC08_L1TP_016042_20211027_20211104_02_T1_B5.TIF\n2021-11-05 08:37:42 75847018 LC08_L1TP_016042_20211027_20211104_02_T1_B6.TIF\n2021-11-05 08:37:44 73697164 LC08_L1TP_016042_20211027_20211104_02_T1_B7.TIF\n2021-11-05 08:37:45 304168225 LC08_L1TP_016042_20211027_20211104_02_T1_B8.TIF\n2021-11-05 08:37:48 60967770 LC08_L1TP_016042_20211027_20211104_02_T1_B9.TIF\n2021-11-05 08:37:49 15036 LC08_L1TP_016042_20211027_20211104_02_T1_MTL.json\n2021-11-05 08:37:49 12369 LC08_L1TP_016042_20211027_20211104_02_T1_MTL.txt\n2021-11-05 08:37:49 18065 LC08_L1TP_016042_20211027_20211104_02_T1_MTL.xml\n2021-11-05 08:37:49 3527153 LC08_L1TP_016042_20211027_20211104_02_T1_QA_PIXEL.TIF\n2021-11-05 08:37:49 222983 LC08_L1TP_016042_20211027_20211104_02_T1_QA_RADSAT.TIF\n2021-11-05 08:37:49 2052489 LC08_L1TP_016042_20211027_20211104_02_T1_SAA.TIF\n2021-11-05 08:37:49 1783702 LC08_L1TP_016042_20211027_20211104_02_T1_SZA.TIF\n2021-11-05 08:37:50 8239344 LC08_L1TP_016042_20211027_20211104_02_T1_VAA.TIF\n2021-11-05 08:37:50 2930826 LC08_L1TP_016042_20211027_20211104_02_T1_VZA.TIF\n2021-11-05 08:37:50 26399 LC08_L1TP_016042_20211027_20211104_02_T1_stac.json\n2021-11-05 08:37:50 46747 LC08_L1TP_016042_20211027_20211104_02_T1_thumb_large.jpeg\n2021-11-05 08:37:50 6015 LC08_L1TP_016042_20211027_20211104_02_T1_thumb_small.jpeg\n\n\nDownload content in bucket to local\n\n! aws s3 cp s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/ . --recursive --request-payer \n\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_ANG.txt to ./LC08_L1TP_016042_20211027_20211104_02_T1_ANG.txt\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B10.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B10.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B2.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B2.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B1.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B1.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B11.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B11.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B6.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B6.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B3.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B3.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B4.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B4.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B5.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B5.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_MTL.json to ./LC08_L1TP_016042_20211027_20211104_02_T1_MTL.json\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_MTL.txt to ./LC08_L1TP_016042_20211027_20211104_02_T1_MTL.txt\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_MTL.xml to ./LC08_L1TP_016042_20211027_20211104_02_T1_MTL.xml\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B7.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B7.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_SZA.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_SZA.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_SAA.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_SAA.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_QA_RADSAT.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_QA_RADSAT.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_QA_PIXEL.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_QA_PIXEL.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_VAA.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_VAA.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_stac.json to ./LC08_L1TP_016042_20211027_20211104_02_T1_stac.json\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_thumb_large.jpeg to ./LC08_L1TP_016042_20211027_20211104_02_T1_thumb_large.jpeg\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_thumb_small.jpeg to ./LC08_L1TP_016042_20211027_20211104_02_T1_thumb_small.jpeg\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B9.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B9.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_VZA.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_VZA.TIF\ndownload: s3://usgs-landsat/collection02/level-1/standard/oli-tirs/2021/016/042/LC08_L1TP_016042_20211027_20211104_02_T1/LC08_L1TP_016042_20211027_20211104_02_T1_B8.TIF to ./LC08_L1TP_016042_20211027_20211104_02_T1_B8.TIF" - }, - { - "objectID": "examples/LPDAAC/Find_and_Access_HLS_PointBuffer.html", - "href": "examples/LPDAAC/Find_and_Access_HLS_PointBuffer.html", - "title": "Create buffer around point location", - "section": "", - "text": "import geopandas\nimport json\nimport shapely\nimport shapely.geometry\nfrom shapely.ops import transform\nfrom shapely.geometry import Point\nfrom shapely.geometry import Polygon\nimport geoviews as gv\nfrom pystac_client import Client \nimport xarray\nimport rasterio as rio\nimport rioxarray\nimport os\ngv.extension('bokeh', 'matplotlib')\nlon = -84.0106\nlat = 10.432\nbuffer = 0.001 # decimal degrees equal to ~222 meters\nloc = Point(lon, lat).buffer(buffer)\ntype(loc)\n\nshapely.geometry.polygon.Polygon\nbase = gv.tile_sources.EsriImagery.opts(width=650, height=500)\nsample_loc = gv.Polygons(loc).opts(line_color='yellow', line_width=10, color=None)\nbase * sample_loc\nTranslate the shapely polygon to a GeoJSON Polygon Object, which is what the STAC API expects.\nroi = json.loads(geopandas.GeoSeries([loc]).to_json())\nroi\n\n{'type': 'FeatureCollection',\n 'features': [{'id': '0',\n 'type': 'Feature',\n 'properties': {},\n 'geometry': {'type': 'Polygon',\n 'coordinates': [[[-84.00959999999999, 10.432],\n [-84.00960481527332, 10.43190198285967],\n [-84.00961921471959, 10.431804909677984],\n [-84.00964305966427, 10.431709715322746],\n [-84.00967612046749, 10.431617316567635],\n [-84.00971807873565, 10.431528603263175],\n [-84.00976853038769, 10.43144442976698],\n [-84.00982698954664, 10.431365606715836],\n [-84.0098928932188, 10.431292893218814],\n [-84.00996560671584, 10.431226989546637],\n [-84.01004442976698, 10.431168530387698],\n [-84.01012860326317, 10.431118078735652],\n [-84.01021731656763, 10.43107612046749],\n [-84.01030971532275, 10.431043059664269],\n [-84.01040490967799, 10.431019214719598],\n [-84.01050198285967, 10.431004815273328],\n [-84.0106, 10.431000000000001],\n [-84.01069801714033, 10.431004815273328],\n [-84.010795090322, 10.431019214719598],\n [-84.01089028467725, 10.431043059664269],\n [-84.01098268343236, 10.43107612046749],\n [-84.01107139673682, 10.431118078735652],\n [-84.01115557023302, 10.431168530387698],\n [-84.01123439328416, 10.431226989546637],\n [-84.01130710678119, 10.431292893218814],\n [-84.01137301045335, 10.431365606715836],\n [-84.0114314696123, 10.43144442976698],\n [-84.01148192126435, 10.431528603263175],\n [-84.0115238795325, 10.431617316567635],\n [-84.01155694033572, 10.431709715322746],\n [-84.0115807852804, 10.431804909677984],\n [-84.01159518472667, 10.43190198285967],\n [-84.0116, 10.432],\n [-84.01159518472667, 10.43209801714033],\n [-84.0115807852804, 10.432195090322017],\n [-84.01155694033572, 10.432290284677254],\n [-84.0115238795325, 10.432382683432365],\n [-84.01148192126435, 10.432471396736826],\n [-84.0114314696123, 10.43255557023302],\n [-84.01137301045335, 10.432634393284165],\n [-84.01130710678119, 10.432707106781187],\n [-84.01123439328416, 10.432773010453364],\n [-84.01115557023302, 10.432831469612303],\n [-84.01107139673682, 10.432881921264348],\n [-84.01098268343236, 10.432923879532511],\n [-84.01089028467725, 10.432956940335732],\n [-84.010795090322, 10.432980785280403],\n [-84.01069801714033, 10.432995184726673],\n [-84.0106, 10.433],\n [-84.01050198285967, 10.432995184726673],\n [-84.01040490967799, 10.432980785280403],\n [-84.01030971532275, 10.432956940335732],\n [-84.01021731656763, 10.432923879532511],\n [-84.01012860326317, 10.432881921264348],\n [-84.01004442976698, 10.432831469612303],\n [-84.00996560671584, 10.432773010453364],\n [-84.0098928932188, 10.432707106781187],\n [-84.00982698954664, 10.432634393284165],\n [-84.00976853038769, 10.43255557023302],\n [-84.00971807873565, 10.432471396736826],\n [-84.00967612046749, 10.432382683432365],\n [-84.00964305966427, 10.432290284677254],\n [-84.00961921471959, 10.432195090322017],\n [-84.00960481527332, 10.43209801714033],\n [-84.00959999999999, 10.432]]]},\n 'bbox': [-84.0116, 10.431000000000001, -84.00959999999999, 10.433]}],\n 'bbox': [-84.0116, 10.431000000000001, -84.00959999999999, 10.433]}\nGeoJSON Polygon Object vertices are expected to be in counterclockwise order. The current object vertices are in clockwise order. Below we reverse the order of the vertices and assign them back to our roi object.\nroi['features'][0]['geometry']['coordinates'][0] = roi['features'][0]['geometry']['coordinates'][0][::-1]" - }, - { - "objectID": "examples/LPDAAC/Find_and_Access_HLS_PointBuffer.html#find-data-using-the-stac-api", - "href": "examples/LPDAAC/Find_and_Access_HLS_PointBuffer.html#find-data-using-the-stac-api", - "title": "Create buffer around point location", - "section": "Find data using the STAC API", - "text": "Find data using the STAC API\nSpecify the STAC API endpoint\n\nSTAC_URL = 'https://cmr.earthdata.nasa.gov/stac'\n\nConnect to the LPCLOUD catalog using PySTAC Client\n\ncatalog = Client.open(f'{STAC_URL}/LPCLOUD/')\n\nSubmit a query for: - HLSL30 and HLSS30 - The roi’s geometry - Dates between 05-01-2021 and 06-30-2022\n\nsearch = catalog.search(\n collections = ['HLSL30.v2.0', 'HLSS30.v2.0'],\n intersects = roi['features'][0]['geometry'],\n datetime = '2021-05/2022-06'\n) \n\n\nsearch.matched()\n\n87\n\n\nGet all the STAC Items/granules\n\nitems = search.get_all_items()\n\n\nlen(items)\n\n87\n\n\nGet the first STAC Item from the list and print off the available assets\n\nitems[0].assets\n\n{'B07': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B07.tif>,\n 'B01': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B01.tif>,\n 'B04': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B04.tif>,\n 'B06': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B06.tif>,\n 'VZA': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.VZA.tif>,\n 'Fmask': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.Fmask.tif>,\n 'B02': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B02.tif>,\n 'SAA': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.SAA.tif>,\n 'B11': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B11.tif>,\n 'B03': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B03.tif>,\n 'VAA': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.VAA.tif>,\n 'B10': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B10.tif>,\n 'B05': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B05.tif>,\n 'B09': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B09.tif>,\n 'SZA': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.SZA.tif>,\n 'browse': <Asset href=https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.jpg>,\n 'metadata': <Asset href=https://cmr.earthdata.nasa.gov/search/concepts/G2144745416-LPCLOUD.xml>}\n\n\nGrab the URL for the B04 asset\n\ndata_url = items[0].assets['B04'].href\ndata_url\n\n'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T16PHS.2021133T155359.v2.0/HLS.L30.T16PHS.2021133T155359.v2.0.B04.tif'\n\n\nNOTE: The above returns the HTTPS URL. For direct S3 access of cloud assets the S3 URL is need. See the following resource on how to generate the S3 URL: https://nasa-openscapes.github.io/2021-Cloud-Hackathon/tutorials/02_Data_Discovery_CMR-STAC_API.html" - }, - { - "objectID": "examples/LPDAAC/Find_and_Access_HLS_PointBuffer.html#read-in-the-url-using-rioxarray", - "href": "examples/LPDAAC/Find_and_Access_HLS_PointBuffer.html#read-in-the-url-using-rioxarray", - "title": "Create buffer around point location", - "section": "Read in the URL using rioxarray", - "text": "Read in the URL using rioxarray\nNOTE: The example below show how to access the asset via HTTPS. An example for performing direct S3 access can be found here: https://nasa-openscapes.github.io/2021-Cloud-Hackathon/tutorials/05_Data_Access_Direct_S3.html\nFirst we need to set our gdal configs. Make sure a .netrc file has been created in your home directory\n\nrio_env = rio.Env(GDAL_DISABLE_READDIR_ON_OPEN='EMPTY',\n GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/cookies.txt'),\n GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/cookies.txt'))\nrio_env.__enter__()\n\n<rasterio.env.Env at 0x7f846be57f70>\n\n\n\nda = rioxarray.open_rasterio(data_url)\n\n\nda\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (band: 1, y: 3660, x: 3660)>\n[13395600 values with dtype=int16]\nCoordinates:\n * band (band) int64 1\n * x (x) float64 8e+05 8e+05 8.001e+05 ... 9.097e+05 9.098e+05\n * y (y) float64 1.2e+06 1.2e+06 1.2e+06 ... 1.09e+06 1.09e+06\n spatial_ref int64 0\nAttributes:\n _FillValue: -9999.0\n scale_factor: 0.0001\n add_offset: 0.0\n long_name: Redxarray.DataArrayband: 1y: 3660x: 3660...[13395600 values with dtype=int16]Coordinates: (4)band(band)int641array([1])x(x)float648e+05 8e+05 ... 9.097e+05 9.098e+05array([799995., 800025., 800055., ..., 909705., 909735., 909765.])y(y)float641.2e+06 1.2e+06 ... 1.09e+06array([1199985., 1199955., 1199925., ..., 1090275., 1090245., 1090215.])spatial_ref()int640crs_wkt :PROJCS[\"UTM Zone 16, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not_specified_based_on_WGS_84_spheroid\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-87],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not_specified_based_on_WGS_84_spheroidprojected_crs_name :UTM Zone 16, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-87.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 16, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not_specified_based_on_WGS_84_spheroid\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-87],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :799980.0 30.0 0.0 1200000.0 0.0 -30.0array(0)Attributes: (4)_FillValue :-9999.0scale_factor :0.0001add_offset :0.0long_name :Red\n\n\nDrop the ‘band’ coordinate\n\nda.squeeze('band', drop=True)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (y: 3660, x: 3660)>\n[13395600 values with dtype=int16]\nCoordinates:\n * x (x) float64 8e+05 8e+05 8.001e+05 ... 9.097e+05 9.098e+05\n * y (y) float64 1.2e+06 1.2e+06 1.2e+06 ... 1.09e+06 1.09e+06\n spatial_ref int64 0\nAttributes:\n _FillValue: -9999.0\n scale_factor: 0.0001\n add_offset: 0.0\n long_name: Redxarray.DataArrayy: 3660x: 3660...[13395600 values with dtype=int16]Coordinates: (3)x(x)float648e+05 8e+05 ... 9.097e+05 9.098e+05array([799995., 800025., 800055., ..., 909705., 909735., 909765.])y(y)float641.2e+06 1.2e+06 ... 1.09e+06array([1199985., 1199955., 1199925., ..., 1090275., 1090245., 1090215.])spatial_ref()int640crs_wkt :PROJCS[\"UTM Zone 16, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not_specified_based_on_WGS_84_spheroid\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-87],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not_specified_based_on_WGS_84_spheroidprojected_crs_name :UTM Zone 16, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-87.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 16, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not_specified_based_on_WGS_84_spheroid\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-87],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :799980.0 30.0 0.0 1200000.0 0.0 -30.0array(0)Attributes: (4)_FillValue :-9999.0scale_factor :0.0001add_offset :0.0long_name :Red" - }, - { - "objectID": "examples/NSIDC/ICESat2-CMR-AWS-S3.html", - "href": "examples/NSIDC/ICESat2-CMR-AWS-S3.html", - "title": "Accessing NSIDC Cloud Collections Using CMR", - "section": "", - "text": "Programmatic access of NSIDC data can happen in 2 ways:" - }, - { - "objectID": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#the-big-picture", - "href": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#the-big-picture", - "title": "Accessing NSIDC Cloud Collections Using CMR", - "section": "The big picture:", - "text": "The big picture:\nThere is nothing wrong with downloading data to our local machine but that can get complicated or even impossible if a dataset is too large. For this reason NSIDC along with other NASA data centers started to collocate or migrate their dataset holdings to the cloud.\n\nSteps\n\nAuthenticate with the NASA Earthdata Login API (EDL).\nSearch granules/collections using a CMR client that supports authentication\nParse CMR responses and get AWS S3 URLs\nAccess the data granules using temporary AWS credentials given by the NSIDC cloud credentials endpoint\n\n\n\nData used:\n\nICESat-2 ATL03: This data set contains height above the WGS 84 ellipsoid (ITRF2014 reference frame), latitude, longitude, and time for all photons.\n\n\n\nRequirements\n\nNASA Eartdata Login (EDL) credentials\npython libraries:\n\nh5py\nmatplotlib\nxarray\ns3fs\npython-cmr\n\ncmr helpers: included in this notebook" - }, - { - "objectID": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#querying-cmr-for-nsidc-data-in-the-cloud", - "href": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#querying-cmr-for-nsidc-data-in-the-cloud", - "title": "Accessing NSIDC Cloud Collections Using CMR", - "section": "Querying CMR for NSIDC data in the cloud", - "text": "Querying CMR for NSIDC data in the cloud\nMost collections at NSIDC have not being migrated to the cloud and can be found using CMR with no authentication at all. Here is a simple example for altimeter data (ATL03) coming from the ICESat-2 mission. First we’ll search the regular collection and then we’ll do the same using the cloud collection.\nNote: This notebook uses CMR to search and locate the data granules, this is not the only workflow for data access and discovery.\n\nHarmonyPy: Uses Harmony the NASA API to search, subset and transform the data in the cloud.\ncmr-stac: A “static” metadata catalog than can be read by Intake oand other client libraries to optimize the access of files in the cloud." - }, - { - "objectID": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#cloud-collections", - "href": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#cloud-collections", - "title": "Accessing NSIDC Cloud Collections Using CMR", - "section": "Cloud Collections", - "text": "Cloud Collections\nSome NSIDC cloud collections are not yet, which means that temporarily you’ll have to request access emailing nsidc@nsidc.org so your Eartdata login is in the authorized list for early users.\n\nimport getpass\nimport textwrap\n\nfrom cmr.search import collection as cmr_collection\nfrom cmr.search import granule \nfrom cmr.auth import token\n\nfrom cmr_auth import CMRAuth\n\n# NON_AWS collections are hosted at the NSIDC DAAC data center\n# AWS_CLOUD collections are hosted at AWS S3 us-west-2\nNSIDC_PROVIDERS = {\n 'NSIDC_HOSTED': 'NSIDC_ECS', \n 'AWS_HOSTED':'NSIDC_CPRD'\n}\n\n# Use your own EDL username\nUSER = 'betolink'\n\nprint('Enter your NASA Earthdata login password:')\npassword = getpass.getpass()\n# This helper class will handle credentials with CMR\nCMR_auth = CMRAuth(USER, password)\n# Token to search preliminary collections on CMR\ncmr_token = CMR_auth.get_token()\n\n\n# The query object uses a simple python dictionary\nquery = {'short_name':'ATL03',\n 'token': cmr_token,\n 'provider': NSIDC_PROVIDERS['AWS_HOSTED']}\n\ncollections = cmr_collection.search(query)\n\nfor collection in collections[0:3]:\n wrapped_abstract = '\\n'.join(textwrap.wrap(f\"Abstract: {collection['umm']['Abstract']}\", 80)) + '\\n'\n print(f\"concept-id: {collection['meta']['concept-id']}\\n\" +\n f\"Title: {collection['umm']['EntryTitle']}\\n\" +\n wrapped_abstract)" - }, - { - "objectID": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#searching-for-data-granules-in-the-cloud-with-cmr", - "href": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#searching-for-data-granules-in-the-cloud-with-cmr", - "title": "Accessing NSIDC Cloud Collections Using CMR", - "section": "Searching for data granules in the cloud with CMR", - "text": "Searching for data granules in the cloud with CMR\nCMR uses different collection id’s for datasets in the cloud.\n\n# now that we have the concept-id for our ATL03 in the cloud we do the same thing we did with ATL03 hosted at\nfrom cmr_serializer import QueryResult\n# NSIDC but using the cloud concept-id\n# Jeneau ice sheet\nquery = {'concept-id': 'C2027878642-NSIDC_CPRD',\n 'token': cmr_token,\n 'bounding_box': '-135.1977,58.3325,-133.3410,58.9839'}\n\n# Querying for ATL03 v3 using its concept-id and a bounding box\nresults = granule.search(query, limit=1000)\ngranules = QueryResult(results).items()\n\nprint(f\"Total granules found: {len(results)} \\n\")\n\n# Print the first 3 granules\nfor g in granules[0:3]:\n display(g)\n # You can use: print(g) for the regular text representation.\n\n\nNOTE: Not all the data granules for NSIDC datasets have been migrated to S3. This might result in different counts between the NSIDC hosted data collections and the ones in AWS S3\n\n\n# We can list the s3 links but \nfor g in granules:\n for link in g.data_links():\n print(link)\n\nWe note that our RelatedLinks array now contain links to AWS S3, these are the direct URIs for our data granules in the AWS us-west-2 region." - }, - { - "objectID": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#data-access-using-aws-s3", - "href": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#data-access-using-aws-s3", - "title": "Accessing NSIDC Cloud Collections Using CMR", - "section": "Data Access using AWS S3", - "text": "Data Access using AWS S3\n\nIMPORTANT: This section will only work if this notebook is running on the AWS us-west-2 zone\n\nThere is more than one way of accessing data on AWS S3, either downloading it to your local machine using the official client library or using a python library.\nPerformance tip: using the HTTPS URLs will decrease the access performance since these links have to internally be processed by AWS’s content delivery system (CloudFront). To get a better performance we should access the S3:// URLs with BOTO3 or a high level S3 enabled library (i.e. S3FS)\nRelated links: * HDF in the Cloud challenges and solutions for scientific data * Cloud Storage (Amazon S3) HDF5 Connector\n\nimport s3fs\nimport h5py\n\nimport xarray as xr\nimport numpy as np\n\nimport matplotlib.pyplot as plt\nimport cartopy.crs as ccrs\nimport cartopy.feature as cfeature\n\n\n# READ only temporary credentials\n# This credentials only last 1 hour.\ns3_cred = CMR_auth.get_s3_credentials()\n\ns3_fs = s3fs.S3FileSystem(key=s3_cred['accessKeyId'],\n secret=s3_cred['secretAccessKey'],\n token=s3_cred['sessionToken'])\n\n# Now you could grab S3 links to your cloud instance (EC2, Hub etc) using:\n# s3_fs.get('s3://SOME_LOCATION/ATL03_20181015124359_02580106_004_01.h5', 'test.h5')\n\nNow that we have the propper credentials in our file mapper, we can access the data within AWS us-west-2.\nIf we are not running this notebook in us-west-2 will get an access denied error" - }, - { - "objectID": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#using-xarray-to-open-files-on-s3", - "href": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#using-xarray-to-open-files-on-s3", - "title": "Accessing NSIDC Cloud Collections Using CMR", - "section": "Using xarray to open files on S3", - "text": "Using xarray to open files on S3\nATL data is complex so xarray doesn’t know how to extract the important bits out of it.\n\nwith s3_fs.open('s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/15/ATL03_20181015124359_02580106_004_01.h5', 'rb') as s3f:\n with h5py.File(s3f, 'r') as f:\n print([key for key in f.keys()])\n gt1l = xr.Dataset({'height': (['x'], f['gt1l']['heights']['h_ph'][:]),\n 'latitude': (['x'], f['gt1l']['heights']['lat_ph'][:]), \n 'longitude': (['x'], f['gt1l']['heights']['lon_ph'][:]),\n 'dist_ph': (['x'], f['gt1l']['heights']['dist_ph_along'][:])})\ngt1l" + "objectID": "tutorials/matlab.html#prerequisites", + "href": "tutorials/matlab.html#prerequisites", + "title": "MATLAB Access Single NASA EarthData L2 NetCDF", + "section": "Prerequisites", + "text": "Prerequisites\nIf you’d rather watch these instructions, there is a short demo video here: https://drive.google.com/drive/folders/1en_SHtjtF4pUt4WU0SzyV16SEegjiwrT?usp=sharing\n\nSave the following function in a text file called loadAWSCredentials.m to your MATLAB working folder or to any folder that is on MATLAB search path.\n\nfunction s3Credentials = loadAWSCredentials(daacCredentialsEndpoint, login, password)\n if nargin < 2 || isempty(login)\n login = getenv('EARTHDATA_LOGIN') ;\n end\n if nargin < 3 || isempty(password)\n password = getenv('EARTHDATA_PASSWORD') ;\n end\n\n % Get S3 credentials from EarthData\n opts = weboptions('ContentType', 'json', 'HeaderFields', ...\n {'Authorization', ['Basic ',matlab.net.base64encode([login,':',password])]});\n s3Credentials = webread(daacCredentialsEndpoint, opts) ;\n\n % Set relevant environment variables with AWS credentials/region\n setenv('AWS_ACCESS_KEY_ID', s3Credentials.accessKeyId) ;\n setenv('AWS_SECRET_ACCESS_KEY', s3Credentials.secretAccessKey) ;\n setenv('AWS_SESSION_TOKEN', s3Credentials.sessionToken) ;\n setenv('AWS_DEFAULT_REGION', 'us-west-2') ;\nend\n\nDefine environment variables that specify your EarthData credentials by executing the following (updated with your credentials):\n\nsetenv(\"EARTHDATA_LOGIN\",\"your_earthdata_login\");\nsetenv(\"EARTHDATA_PASSWORD\",\"your_earthdata_password\");\nThis can be done manually in the command window, in your MATLAB startup script, or in any MATLAB function/script that is executed before calling loadAWSCredentials. We recommend not to do it from your project main script, so you can share it without divulging your credentials.\n\nSearch NASA EarthData and find the S3 link you want to access." }, { - "objectID": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#plotting-the-data", - "href": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#plotting-the-data", - "title": "Accessing NSIDC Cloud Collections Using CMR", - "section": "Plotting the data", - "text": "Plotting the data\n\ngt1l.height.plot()\n\n\n%matplotlib widget\n\nfig, ax = plt.subplots(figsize=(14, 4))\n\ngt1l.height.plot(ax=ax, ls='', marker='o', ms=1)\ngt1l.height.rolling(x=1000, min_periods=500, center=True).mean().plot(ax=ax, c='k', lw=2)\nax.set_xlabel('Along track distance (m)', fontsize=12);\nax.set_ylabel('Photon Height (m)', fontsize=12)\nax.set_title('ICESat-2 ATL03', fontsize=14)\nax.tick_params(axis='both', which='major', labelsize=12)\n\nsubax = fig.add_axes([0.69,0.50,0.3,0.3], projection=ccrs.NorthPolarStereo())\nsubax.set_aspect('equal')\nsubax.set_extent([-180., 180., 30, 90.], ccrs.PlateCarree())\nsubax.add_feature(cfeature.LAND)\nsubax.plot(gt1l.longitude, gt1l.latitude, transform=ccrs.PlateCarree(), lw=1);\n\nfig.savefig('test.png')\n\nplt.show()" + "objectID": "tutorials/matlab.html#accessing-data", + "href": "tutorials/matlab.html#accessing-data", + "title": "MATLAB Access Single NASA EarthData L2 NetCDF", + "section": "Accessing data", + "text": "Accessing data\nMATLAB HDF5 interfaces can access netCDF4 files stored in S3/Azure, while the netCDF interfaces cannot. The example below shows how to access a netCDF file using the HDF5 interfaces. SEe also this video: MATLAB and HDF5: Compression, Cloud, and Community – Ellen Johnson, MathWorks.\nMATLAB provides two interfaces for accessing HDF5 datasets:\n\nHigh-level access functions make it easy to read and view data in an HDF5 file or write a variable from the MATLAB® workspace into an HDF5 file\nLow-level functions in the HDF5 library packages provide direct access to the more than 300 functions in the HDF5 C library from the HDF Group.\n\nLow-level functions allow you to access advanced HDF5 functionality and bypasses I/O overhead in the high-level interface, but has a substantially higher learning curve unless already familiar with the HDF5 C API.\n\nExample\n\nFind the relevant DAAC/endpoint for the dataset that you want to access. For the following dataset:\n\ns3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20100619062008-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\nThe relevant DAAC is PODAAC and the relevant endpoint is:\nhttps://archive.podaac.earthdata.nasa.gov/s3credentials\n\nSet AWS credentials using the loadAWSCredentials function, passing the relevant endpoint:\n\ndaacCredentialsEndpoint = \"https://archive.podaac.earthdata.nasa.gov/s3credentials\";\nloadAWSCredentials(daacCredentialsEndpoint);\nThese credentials are temporary; they expire every 30 minutes or so and must be refreshed periodically.\n\nAccess data Define path of relevant file and dataset:\n\nFILE_NAME = \"s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20100619062008-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\";\nDATASET_NAME = \"sea_surface_temperature_4um\";\nDATASET_PATH = \"/\" + DATASET_NAME ;\nFirst, check that you can access the dataset:\nh5disp(FILE_NAME);\nIf credentials and paths are correct, h5disp will display the structure of the data file.\n\n\nUsing MATLAB high-level HDF5 interface\nunits = h5readatt(FILE_NAME,DATASET_PATH,\"units\");\nfill_value = h5readatt(FILE_NAME,DATASET_PATH,\"_FillValue\");\nname = h5readatt(FILE_NAME,DATASET_PATH,\"long_name\");\ndata = h5read(FILE_NAME,DATASET_PATH);\ndata(data == fill_value) = NaN;\nlat = h5read(FILE_NAME,\"/lat\");\nlon = h5read(FILE_NAME,\"/lon\");\nBuild a map using contour:\ncontour(lon,lat,data);\ntitle(sprintf(\"%s [%s]\",name,units));\n\n\n\nMATLAB example map\n\n\n\n\nUsing MATLAB low-level HDF5 interface\n% Open file and root group\nfile_id = H5F.open(FILE_NAME,\"H5F_ACC_RDONLY\",\"H5P_DEFAULT\");\nroot_id = H5G.open(file_id,\"/\") ;\n\n% Open/read sea surface temperature (SST) dataset\ndata_id = H5D.open(file_id,DATASET_PATH);\ndata=H5D.read(data_id,\"H5T_NATIVE_DOUBLE\",\"H5S_ALL\",\"H5S_ALL\",\"H5P_DEFAULT\");\n\n% Open/read/close SST units attribute\nattr_id = H5A.open_by_name(root_id,DATASET_NAME,\"units\");\nunits = H5A.read(attr_id,\"H5ML_DEFAULT\");\nH5A.close(attr_id);\n\n% Open/read/close SST long_name attribute\nattr_id = H5A.open_by_name(root_id,DATASET_NAME,\"long_name\");\nlong_name = H5A.read(attr_id,\"H5ML_DEFAULT\");\nH5A.close(attr_id);\n\n% Open/read/close SST _FillValue attribute\nattr_id = H5A.open_by_name(root_id,DATASET_NAME,\"_FillValue\");\nfillvalue = H5A.read(attr_id,\"H5T_NATIVE_DOUBLE\");\nH5A.close(attr_id);\n\n% Close SST dataset and root group\nH5D.close(data_id);\nH5G.close(root_id);\n\n% Open/read/close latitude dataset\nlat_id = H5D.open(file_id,\"lat\");\nlat = H5D.read(lat_id,\"H5T_NATIVE_DOUBLE\",\"H5S_ALL\",\"H5S_ALL\",\"H5P_DEFAULT\");\nH5D.close(lat_id);\n\n% Open/read/close longitude dataset\nlon_id = H5D.open(file_id,\"lon\");\nlon = H5D.read(lon_id,\"H5T_NATIVE_DOUBLE\",\"H5S_ALL\",\"H5S_ALL\",\"H5P_DEFAULT\");\nH5D.close(lon_id);\n\n% Close file\nH5F.close(file_id);\n\n% Replace fill value with NaN\ndata(data==fillvalue) = NaN;\n\n% Plot contour map\ncontour(lon,lat,data);\ntitle(sprintf(\"%s [%s]\",long_name{1},units{1}));" }, { - "objectID": "examples/Transform-workflow.html", - "href": "examples/Transform-workflow.html", - "title": "Earthdata Cloud Exploration: Direct Access and Harmony workflow", - "section": "", - "text": "This notebook will walk through two different access and transformation options across a single dataset available in the Earthdata Cloud: Direct s3 access and Harmony transformation services. These steps are based off of several notebook tutorials presented during the 2021 Earthdata Cloud Hackathon and 2021 AGU Workshop.\nThe following use case focuses on Sea Surface Temperature (SST) from the GHRSST MODIS L2 dataset (MODIS_T-JPL-L2P-v2019.0) available from PO.DAAC in the Earthdata Cloud.\n\n\nUsing NASA Worldview, we can first explore SST during a tropical storm event; in this case, we can overlay L2 and L4 SST variables against true color imagery to observe Hurricane Ida in August 2021. Although this is a small sample set, this use case could be expanded to explore how SST responds during the Atlantic hurricane over the next several months. The same data that we are requesting below using direct s3 access and via Harmony-py can also be requested using NASA Earthdata Search\n\n\n\nHurrican Ida snapshot - Worldview\n\n\nWe will access a single netCDF file from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataset. This approach leverages S3 native protocols for efficient access to the data.\nAfter inspecting the data, we will then access the same file, but in this case we will also subset the data to our area of interest using Harmony." + "objectID": "tutorials/matlab.html#launch-with-aws-console", + "href": "tutorials/matlab.html#launch-with-aws-console", + "title": "MATLAB Access Single NASA EarthData L2 NetCDF", + "section": "Launch with AWS Console", + "text": "Launch with AWS Console\nAn alternative is to launch with the AWS Console. This chapter is for research teams currently working in MATLAB with NASA EarthData and wanting to take advantage of doing analysis in the Cloud. Our initial focus is on Amazon Web Services (AWS). For general, background on the Cloud and NASA Earthdata’s migration to the Cloud, checkout earlier chapters of the cookbook [TO DO: Add a link and specific reference].\n\nPrerequisites\n\nMATLAB License - You must have access to a MATLAB license that allows for access to the cloud. To check your license, in the MATLAB command line enter:\nver -support\nAWS Account - For the current experiments, we will access the AWS console, so you need an AWS account. It is free to set up, but will require a credit card linked to the account. https://aws.amazon.com/\nAWS Region: We are working in AWS US-West-2 because NASA Earthdata is hosted in US-West-2. If there is an option to pick an AWS region, pick US-West-2.\nGitHub Account - If you want to use GitHub as part of your workflow and don’t already have an account, create a GitHub account: https://github.com/\nNASA Earthdata login - Create an account here: https://urs.earthdata.nasa.gov/users/new\nMicrosoft Remote Desktop - When we spin up the AWS virtual machine with MATLAB, we will need to have a remote desktop option. These how-to’s were done on a mac and we tested the free Microsoft Remote Desktop. Download for macs\n\n\n\nPrerequisite: AWS EC2 Key Pair\nThis isn’t included in the numbered prereq list above because it is a bit more involved than creating an account or knowing what region we work in. You need to create an SSH Key Pair in the region you want to work in. This is a one-time step that you won’t need to do each time you launch the stack.\n\nLog in to AWS\nIn the search box enter, ‘Create AWS Key Pair.’ Under features choose ‘Key Pairs’ (AWS help)\n\nCheck that the top bar says, ‘Oregon’ (AKA - US-West-2) and if not, click that down arrow to choose US-West-2. Then click the orange ‘Create key pair’ button.\n\nTroubleshooting tip: If you create the key outside of the region you want to work in, it will not show up when you launch the stack below.\nOn the form - give your key a name, choose the RSA and PEM options (these should be the defaults.)\n\nOk - now we are ready to start!\n\n\nCreating the AWS Stack with MATLAB\nNote: The first time I launched the AWS Stack it seemed to take for-ever. I thought “how could I possibly need to do this every time I want to use MATLAB in the cloud?” It does speed up, eventually it get’s a bit faster. I also have learned to plan a bit better - if I know I want to do some work, I get the launched, have coffee, and when I come back it’s ready to roll.\n\nFrom this MATLAB GitHub page click the release for 2022a under deployment steps.\nThis brings up a list of MATLAB on Amazon Web Services (Linux VM). Choose & click the ‘launch stack’ link for US-West-2.\n\nThis opens the ‘Quick create stack’ form based on the MATLAB template. That means that when you launch this stack it will come with MATLAB on the desktop. Fill out the form to create the AWS stack:\n\nGive the stack a name like ‘matlab-test’\nKeep the pre-filled options the same for now.\nRemote Access:\n\n“Allow Connections From:” You will need to know your IP address. You can google, “what’s my IP address?”\nEnter your IP address followed by a /32 like this -> [my.IP.address/32]\nIn the SSH Key Pair - the key pair you created above should show up in the drop down. If it doesn’t show up, see the troubleshooting tip.\nPick a remote password. This is not your AWS password or your GitHub password, this is the password that you will use to login with the microsoft remote desktop (username: ubuntu)\n\nNetwork configuration\n\nThere is one VPC option - choose that\nFor subnet - I pick the first one and it works. So pick the first option.\n\nAutoshutdown hasn’t worked for me so far, so for now I leave this set as never and delete the stack when I am finished.\nCheck the box that “I acknowledge that AWS CloudFormation might create IAM resources.”\nClick ‘Create stack’\nWait…. [~ 10 minutes]\n\nYou can check the status by clicking the refresh button on the right corner\n\n\n\n\nLaunch the AWS Stack with Microsoft Remote Desktop\n\nOnce the stack is created it will say ‘Create_complete’ on the left side.\n\nClick the outputs tab and copy the value text. It will start with ‘ec2-…’\nOpen Microsoft Remote Desktop\n\nClick the + to add a PC\nPaste the value text as the PC Name\nClick on the grey box of your new computer in the remote desktop window\nA login will pop up\n\n\nUsername is ubuntu\nPassword is the password you set in 3.3.3 above in the section on ‘Creating AWS Stack’\n\nA certificate message will pop up - say ok\nThe desktop will launch\n\nWait … [~2 mins]\n\n\n\nOpen MATLAB on Remote Desktop\n\nClick the MATLAB icon on the remote desktop\n\nWait … [~4 mins]\nLogin with your MATLAB credentials\nYou are in!\n\n\n\nShutdown your AWS Stack\nAfter each session you need to turn off the AWS Stack. If you forget this step and leave it running it is like keeping a computer on for the month. For the large instance it costs $0.5/day so it’s a few dollars a month.\n\nGo back to AWS\nSearch for stack\nClick on the name of your stack\nClick ‘Delete’\n\nConfirm the delete" }, { - "objectID": "examples/Transform-workflow.html#summary", - "href": "examples/Transform-workflow.html#summary", - "title": "Earthdata Cloud Exploration: Direct Access and Harmony workflow", - "section": "", - "text": "This notebook will walk through two different access and transformation options across a single dataset available in the Earthdata Cloud: Direct s3 access and Harmony transformation services. These steps are based off of several notebook tutorials presented during the 2021 Earthdata Cloud Hackathon and 2021 AGU Workshop.\nThe following use case focuses on Sea Surface Temperature (SST) from the GHRSST MODIS L2 dataset (MODIS_T-JPL-L2P-v2019.0) available from PO.DAAC in the Earthdata Cloud.\n\n\nUsing NASA Worldview, we can first explore SST during a tropical storm event; in this case, we can overlay L2 and L4 SST variables against true color imagery to observe Hurricane Ida in August 2021. Although this is a small sample set, this use case could be expanded to explore how SST responds during the Atlantic hurricane over the next several months. The same data that we are requesting below using direct s3 access and via Harmony-py can also be requested using NASA Earthdata Search\n\n\n\nHurrican Ida snapshot - Worldview\n\n\nWe will access a single netCDF file from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataset. This approach leverages S3 native protocols for efficient access to the data.\nAfter inspecting the data, we will then access the same file, but in this case we will also subset the data to our area of interest using Harmony." + "objectID": "tutorials/04_On-Prem_Cloud.html#summary", + "href": "tutorials/04_On-Prem_Cloud.html#summary", + "title": "Pairing Cloud and non-Cloud Data", + "section": "Summary", + "text": "Summary\n\nThis tutorial will combine several workflow steps and components from the previous days, demonstrating the process of using the geolocation of data available outside of the Earthdata Cloud to then access coincident variables of cloud-accessible data. This may be a common use case as NASA Earthdata continues to migrate to the cloud, producing a “hybrid” data archive across Amazon Web Services (AWS) and original on-premise data storage systems. Additionally, you may also want to combine field measurements with remote sensing data available on the Earthdata Cloud.\nThis specific example explores the pairing of the ICESat-2 ATL07 Sea Ice Height data product, currently (as of November 2021) available publicly via direct download at the NSIDC DAAC, along with Sea Surface Temperature (SST) from the GHRSST MODIS L2 dataset (MODIS_A-JPL-L2P-v2019.0) available from PO.DAAC on the Earthdata Cloud.\nThe use case we’re looking at today centers over an area north of Greenland for a single day in June, where a melt pond was observed using the NASA OpenAltimetry application. Melt ponds are an important feature of Arctic sea ice dynamics, leading to an decrease in sea ice albedo and other changes in heat balance. Many NASA Earthdata datasets produce variables including sea ice albedo, sea surface temperature, air temperature, and sea ice height, which can be used to better understand these dynamics." }, { - "objectID": "examples/Transform-workflow.html#import-required-packages", - "href": "examples/Transform-workflow.html#import-required-packages", - "title": "Earthdata Cloud Exploration: Direct Access and Harmony workflow", - "section": "Import Required Packages", - "text": "Import Required Packages\n\n# Earthdata Login\nfrom netrc import netrc\nfrom subprocess import Popen\nfrom platform import system\nfrom getpass import getpass\nimport os\n\n# Direct access\nimport requests\nimport s3fs\nimport xarray as xr\nimport hvplot.xarray\n\n# Harmony\nfrom harmony import BBox, Client, Collection, Request, LinkType\nfrom harmony.config import Environment\nfrom pprint import pprint\nimport datetime as dt" + "objectID": "tutorials/04_On-Prem_Cloud.html#requirements", + "href": "tutorials/04_On-Prem_Cloud.html#requirements", + "title": "Pairing Cloud and non-Cloud Data", + "section": "Requirements", + "text": "Requirements\n\n\nAWS instance running in us-west 2\nEarthdata Login\n.netrc file" }, { - "objectID": "examples/Transform-workflow.html#authentication-for-nasa-earthdata", - "href": "examples/Transform-workflow.html#authentication-for-nasa-earthdata", - "title": "Earthdata Cloud Exploration: Direct Access and Harmony workflow", - "section": "1. Authentication for NASA Earthdata", - "text": "1. Authentication for NASA Earthdata\nBased off of https://nasa-openscapes.github.io/2021-Cloud-Workshop-AGU/tutorials/02_NASA_Earthdata_Authentication.html\nFirst, we will create a hidden .netrc file (_netrc for Window OS) with Earthdata login credentials in your home directory. This file is needed to access NASA Earthdata assets from a scripting environment like Python.\n\nEarthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\nAuthentication via netrc File\nYou will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. An example of the required content is below.\nmachine urs.earthdata.nasa.gov\nlogin <USERNAME>\npassword <PASSWORD>\n<USERNAME> and <PASSWORD> would be replaced by your actual Earthdata Login username and password respectively.\nThe code below will:\n\ncheck what operating system (OS) is being used to determine which netrc file to check for/create (.netrc or _netrc)\ncheck if you have an netrc file, and if so, varify if those credentials are for the Earthdata endpoint\ncreate a netrc file if a netrc file is not present.\n\n\nurs = 'urs.earthdata.nasa.gov' # Earthdata URL endpoint for authentication\nprompts = ['Enter NASA Earthdata Login Username: ',\n 'Enter NASA Earthdata Login Password: ']\n\n# Determine the OS (Windows machines usually use an '_netrc' file)\nnetrc_name = \"_netrc\" if system()==\"Windows\" else \".netrc\"\n\n# Determine if netrc file exists, and if so, if it includes NASA Earthdata Login Credentials\ntry:\n netrcDir = os.path.expanduser(f\"~/{netrc_name}\")\n netrc(netrcDir).authenticators(urs)[0]\n\n# Below, create a netrc file and prompt user for NASA Earthdata Login Username and Password\nexcept FileNotFoundError:\n homeDir = os.path.expanduser(\"~\")\n Popen('touch {0}{2} | echo machine {1} >> {0}{2}'.format(homeDir + os.sep, urs, netrc_name), shell=True)\n Popen('echo login {} >> {}{}'.format(getpass(prompt=prompts[0]), homeDir + os.sep, netrc_name), shell=True)\n Popen('echo \\'password {} \\'>> {}{}'.format(getpass(prompt=prompts[1]), homeDir + os.sep, netrc_name), shell=True)\n # Set restrictive permissions\n Popen('chmod 0600 {0}{1}'.format(homeDir + os.sep, netrc_name), shell=True)\n\n # Determine OS and edit netrc file if it exists but is not set up for NASA Earthdata Login\nexcept TypeError:\n homeDir = os.path.expanduser(\"~\")\n Popen('echo machine {1} >> {0}{2}'.format(homeDir + os.sep, urs, netrc_name), shell=True)\n Popen('echo login {} >> {}{}'.format(getpass(prompt=prompts[0]), homeDir + os.sep, netrc_name), shell=True)\n Popen('echo \\'password {} \\'>> {}{}'.format(getpass(prompt=prompts[1]), homeDir + os.sep, netrc_name), shell=True)\n\n\nSee if the file was created\nIf the file was created, we’ll see a .netrc file (_netrc for Window OS) in the list printed below. To view the contents from a Jupyter environment, click File on the top toolbar, select Open from Path…, type .netrc, and click Open. The .netrc file will open within the text editor.\n\n!!! Beware, your password will be visible if the .netrc file is opened in the text editor.\n\n\n!ls -al ~/" + "objectID": "tutorials/04_On-Prem_Cloud.html#learning-objectives", + "href": "tutorials/04_On-Prem_Cloud.html#learning-objectives", + "title": "Pairing Cloud and non-Cloud Data", + "section": "Learning Objectives", + "text": "Learning Objectives\n\n\nSearch for data programmatically using the Common Metadata Repository (CMR), determining granule (file) coverage across two datasets over an area of interest.\nDownload data from an on-premise storage system to our cloud environment.\nRead in 1-dimensional trajectory data (ICESat-2 ATL07) into xarray and perform attribute conversions.\nSelect and read in sea surface temperature (SST) data (MODIS_A-JPL-L2P-v2019.0) from the Earthdata Cloud into xarray.\nExtract, resample, and plot coincident SST data based on ICESat-2 geolocation." }, { - "objectID": "examples/Transform-workflow.html#accessing-a-netcdf4hdf5-file---s3-direct-access", - "href": "examples/Transform-workflow.html#accessing-a-netcdf4hdf5-file---s3-direct-access", - "title": "Earthdata Cloud Exploration: Direct Access and Harmony workflow", - "section": "2. Accessing a NetCDF4/HDF5 File - S3 Direct Access", - "text": "2. Accessing a NetCDF4/HDF5 File - S3 Direct Access\n_Based off of https://nasa-openscapes.github.io/2021-Cloud-Workshop-AGU/how-tos/Earthdata_Cloud__Single_File_Direct_S3_Access_NetCDF4_Example.html and https://nasa-openscapes.github.io/2021-Cloud-Workshop-AGU/tutorials/04_On-Prem_Cloud.html\n\nSummary\nWe will access a single netCDF file from the GHRSST MODIS L2 dataset (MODIS_T-JPL-L2P-v2019.0) inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataset. This approach leverages S3 native protocols for efficient access to the data.\n\n\nRequirements\n\n1. AWS instance running in us-west-2\nNASA Earthdata Cloud data in S3 can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region.\n\n\n2. Earthdata Login\n\n\n3. netrc File\n\n\n\nLearning Objectives\n\nhow to retrieve temporary S3 credentials for in-region direct S3 bucket access\nhow to define a dataset of interest and find netCDF files in S3 bucket\nhow to perform in-region direct access of MODIS_T-JPL-L2P-v2019.0 data in S3\nhow to plot the data\n\n\n\nGet Temporary AWS Credentials\nDirect S3 access is achieved by passing NASA supplied temporary credentials to AWS so we can interact with S3 objects from applicable Earthdata Cloud buckets. For now, each NASA DAAC has different AWS credentials endpoints. Below are some of the credential endpoints to various DAACs:\n\ns3_cred_endpoint = {\n 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n 'gesdisc': 'https://data.gesdisc.earthdata.nasa.gov/s3credentials',\n 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n 'ornldaac': 'https://data.ornldaac.earthdata.nasa.gov/s3credentials',\n 'ghrcdaac': 'https://data.ghrc.earthdata.nasa.gov/s3credentials'\n}\n\nCreate a function to make a request to an endpoint for temporary credentials. Remember, each DAAC has their own endpoint and credentials are not usable for cloud data from other DAACs.\n\ndef get_temp_creds(provider):\n return requests.get(s3_cred_endpoint[provider]).json()\n\n\ntemp_creds_req = get_temp_creds('podaac')\n#temp_creds_req\n\n\n\nSet up an s3fs session for Direct Access\ns3fs sessions are used for authenticated access to s3 buckets and allow for typical file-system style operations. Below we create session by passing in the temporary credentials we recieved from our temporary credentials endpoint.\n\nfs_s3 = s3fs.S3FileSystem(anon=False, \n key=temp_creds_req['accessKeyId'], \n secret=temp_creds_req['secretAccessKey'], \n token=temp_creds_req['sessionToken'],\n client_kwargs={'region_name':'us-west-2'})\n\nIn this example we’re interested in the GHRSST MODIS L2 collection from NASA’s PO.DAAC in Earthdata Cloud. Below we specify the s3 URL to the data asset in Earthdata Cloud. This URL can be found via Earthdata Search or programmatically through the CMR and CMR-STAC APIs.\n\nshort_name = 'MODIS_T-JPL-L2P-v2019.0'\ns3_url = 's3://podaac-ops-cumulus-protected/MODIS_T-JPL-L2P-v2019.0/20210820033500-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0.nc'\n\n\n\nDirect In-region Access\nOpen with the netCDF file using the s3fs package, then load the cloud asset into a concatenated xarray dataset.\n\ns3_file_obj = fs_s3.open(s3_url, mode='rb')\n\n\nssh_ds = xr.open_dataset(s3_file_obj)\nssh_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (nj: 2030, ni: 1354, time: 1)\nCoordinates:\n lat (nj, ni) float32 ...\n lon (nj, ni) float32 ...\n * time (time) datetime64[ns] 2021-08-20T03:35:00\nDimensions without coordinates: nj, ni\nData variables:\n sea_surface_temperature (time, nj, ni) float32 ...\n sst_dtime (time, nj, ni) timedelta64[ns] ...\n quality_level (time, nj, ni) float32 ...\n sses_bias (time, nj, ni) float32 ...\n sses_standard_deviation (time, nj, ni) float32 ...\n l2p_flags (time, nj, ni) int16 ...\n sea_surface_temperature_4um (time, nj, ni) float32 ...\n quality_level_4um (time, nj, ni) float32 ...\n sses_bias_4um (time, nj, ni) float32 ...\n sses_standard_deviation_4um (time, nj, ni) float32 ...\n wind_speed (time, nj, ni) float32 ...\n dt_analysis (time, nj, ni) float32 ...\nAttributes: (12/49)\n Conventions: CF-1.7, ACDD-1.3\n title: MODIS Terra L2P SST\n summary: Sea surface temperature retrievals produced a...\n references: GHRSST Data Processing Specification v2r5\n institution: NASA/JPL/OBPG/RSMAS\n history: MODIS L2P created at JPL PO.DAAC\n ... ...\n publisher_email: ghrsst-po@nceo.ac.uk\n processing_level: L2P\n cdm_data_type: swath\n startDirection: Ascending\n endDirection: Ascending\n day_night_flag: Nightxarray.DatasetDimensions:nj: 2030ni: 1354time: 1Coordinates: (3)lat(nj, ni)float32...long_name :latitudestandard_name :latitudeunits :degrees_northvalid_min :-90.0valid_max :90.0comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[2748620 values with dtype=float32]lon(nj, ni)float32...long_name :longitudestandard_name :longitudeunits :degrees_eastvalid_min :-180.0valid_max :180.0comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[2748620 values with dtype=float32]time(time)datetime64[ns]2021-08-20T03:35:00long_name :reference time of sst filestandard_name :timecomment :time of first sensor observationcoverage_content_type :coordinatearray(['2021-08-20T03:35:00.000000000'], dtype='datetime64[ns]')Data variables: (12)sea_surface_temperature(time, nj, ni)float32...long_name :sea surface temperaturestandard_name :sea_surface_skin_temperatureunits :kelvinvalid_min :-1000valid_max :10000comment :sea surface temperature from thermal IR (11 um) channelssource :NASA and University of Miamicoverage_content_type :physicalMeasurement[2748620 values with dtype=float32]sst_dtime(time, nj, ni)timedelta64[ns]...long_name :time difference from reference timevalid_min :-32767valid_max :32767comment :time plus sst_dtime gives seconds after 00:00:00 UTC January 1, 1981coverage_content_type :referenceInformation[2748620 values with dtype=timedelta64[ns]]quality_level(time, nj, ni)float32...long_name :quality level of SST pixelvalid_min :0valid_max :5comment :thermal IR SST proximity confidence value; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valueflag_values :[0 1 2 3 4 5]flag_meanings :no_data bad_data worst_quality low_quality acceptable_quality best_qualitycoverage_content_type :qualityInformation[2748620 values with dtype=float32]sses_bias(time, nj, ni)float32...long_name :SSES bias error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :thermal IR SST bias error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]sses_standard_deviation(time, nj, ni)float32...long_name :SSES standard deviation error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :thermal IR SST standard deviation error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]l2p_flags(time, nj, ni)int16...long_name :L2P flagsvalid_min :0valid_max :16comment :These flags can be used to further filter data variablesflag_meanings :microwave land ice lake riverflag_masks :[ 1 2 4 8 16]coverage_content_type :qualityInformation[2748620 values with dtype=int16]sea_surface_temperature_4um(time, nj, ni)float32...long_name :sea surface temperatureunits :kelvinvalid_min :-1000valid_max :10000comment :sea surface temperature from mid-IR (4 um) channels; non L2P core fieldcoverage_content_type :physicalMeasurement[2748620 values with dtype=float32]quality_level_4um(time, nj, ni)float32...long_name :quality level of SST pixelvalid_min :0valid_max :5comment :mid-IR SST proximity confidence value; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valueflag_values :[0 1 2 3 4 5]flag_meanings :no_data bad_data worst_quality low_quality acceptable_quality best_qualitycoverage_content_type :qualityInformation[2748620 values with dtype=float32]sses_bias_4um(time, nj, ni)float32...long_name :SSES bias error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :mid-IR SST bias error; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]sses_standard_deviation_4um(time, nj, ni)float32...long_name :SSES standard deviation error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :mid-IR SST standard deviation error; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]wind_speed(time, nj, ni)float32...long_name :10m wind speedstandard_name :wind_speedunits :m s-1valid_min :-127valid_max :127comment :Wind at 10 meters above the sea surfacesource :TBD. Placeholder. Currently emptygrid_mapping :TBDtime_offset :2.0height :10 mcoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]dt_analysis(time, nj, ni)float32...long_name :deviation from SST reference climatologyunits :kelvinvalid_min :-127valid_max :127comment :TBDsource :TBD. Placeholder. Currently emptycoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]Attributes: (49)Conventions :CF-1.7, ACDD-1.3title :MODIS Terra L2P SSTsummary :Sea surface temperature retrievals produced at the NASA OBPG for the MODIS Aqua sensor. These have been reformatted to GHRSST GDS specifications by the JPL PO.DAACreferences :GHRSST Data Processing Specification v2r5institution :NASA/JPL/OBPG/RSMAShistory :MODIS L2P created at JPL PO.DAACcomment :L2P Core without DT analysis or other ancillary fields; Night, Start Node:Ascending, End Node:Ascending; WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value; Quicklooklicense :GHRSST and PO.DAAC protocol allow data use as free and open.id :MODIS_T-JPL-L2P-v2019.0naming_authority :org.ghrsstproduct_version :2019.0uuid :f6e1f61d-c4a4-4c17-8354-0c15e12d688bgds_version_id :2.0netcdf_version_id :4.1date_created :20210820T153042Zfile_quality_level :3spatial_resolution :1kmstart_time :20210820T033500Ztime_coverage_start :20210820T033500Zstop_time :20210820T033959Ztime_coverage_end :20210820T033959Znorthernmost_latitude :34.0296southernmost_latitude :12.9895easternmost_longitude :-68.4372westernmost_longitude :-95.5518source :MODIS sea surface temperature observations for the OBPGplatform :Terrasensor :MODISmetadata_link :http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MODIS_T-JPL-L2P-v2019.0keywords :Oceans > Ocean Temperature > Sea Surface Temperaturekeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordsstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventiongeospatial_lat_units :degrees_northgeospatial_lat_resolution :0.01geospatial_lon_units :degrees_eastgeospatial_lon_resolution :0.01acknowledgment :The MODIS L2P sea surface temperature data are sponsored by NASAcreator_name :Ed Armstrong, JPL PO.DAACcreator_email :edward.m.armstrong@jpl.nasa.govcreator_url :http://podaac.jpl.nasa.govproject :Group for High Resolution Sea Surface Temperaturepublisher_name :The GHRSST Project Officepublisher_url :http://www.ghrsst.orgpublisher_email :ghrsst-po@nceo.ac.ukprocessing_level :L2Pcdm_data_type :swathstartDirection :AscendingendDirection :Ascendingday_night_flag :Night\n\n\nLet’s make a quick plot to take a look at the sea_surface_temperature variable.\n\nssh_ds.sea_surface_temperature.plot() ;" + "objectID": "tutorials/04_On-Prem_Cloud.html#import-packages", + "href": "tutorials/04_On-Prem_Cloud.html#import-packages", + "title": "Pairing Cloud and non-Cloud Data", + "section": "Import packages", + "text": "Import packages\n\nimport os\nfrom pathlib import Path\nfrom pprint import pprint\n\n# Access via download\nimport requests\n\n# Access AWS S3\nimport s3fs\n\n# Read and work with datasets\nimport xarray as xr\nimport numpy as np\nimport h5py\n\n# For plotting\nimport matplotlib.pyplot as plt\nimport cartopy.crs as ccrs\nfrom shapely.geometry import box\n\n# For resampling\nimport pyresample" }, { - "objectID": "examples/Transform-workflow.html#accessing-multiple-netcdf4hdf5-files---data-subsetting-and-transformation-services-in-the-cloud", - "href": "examples/Transform-workflow.html#accessing-multiple-netcdf4hdf5-files---data-subsetting-and-transformation-services-in-the-cloud", - "title": "Earthdata Cloud Exploration: Direct Access and Harmony workflow", - "section": "3. Accessing Multiple NetCDF4/HDF5 Files - Data Subsetting and Transformation Services in the Cloud", - "text": "3. Accessing Multiple NetCDF4/HDF5 Files - Data Subsetting and Transformation Services in the Cloud\n\nUsing the Harmony-Py library to access customized data from NASA Earthdata\n\nWhat other access options or services exist for this dataset?\nMaybe we’re interested in creating a time series over a larger area or with a larger dataset. Let’s see whether there are other services out there that could either make this operation more performant in the cloud (with a cloud-optimized output) or subsetting services to reduce the data volume.\nBased off of https://nasa-openscapes.github.io/2021-Cloud-Hackathon/tutorials/07_Harmony_Subsetting.html\n\n\n\nSummary\nWe have already explored direct access to the NASA EOSDIS archive in the cloud via the Amazon Simple Storage Service (S3) by using the Common Metadata Repository (CMR) to search for granule locations. In addition to directly accessing the files archived and distributed by each of the NASA DAACs, many datasets also support services that allow us to customize the data via subsetting, reformatting, reprojection, and other transformations.\nThis tutorial demonstrates how to find, request, and use customized data from a new ecosystem of services operating within the NASA Earthdata Cloud: NASA Harmony.\n\n\nBenefits\nBut first, why use this option when we’ve already learned how to access data directly from the NASA Earthdata Cloud?\n\nConsistent access patterns to EOSDIS holdings make cross-data center data access easier\nData reduction services allow us to request only the data we want, in the format and projection we want\nAnalysis Ready Data and cloud access will help reduce time-to-science\nCommunity Development helps reduce the barriers for re-use of code and sharing of domain knowledge\n\nSee more on the Earthdata Harmony landing page, including documentation on the Harmony API itself.\n\n\nObjectives\n\nConceptualize the data transformation service types and offerings provided by NASA Earthdata, including Harmony.\nPractice skills learned from the introductory CMR tutorial to discover what access and service options exist for a given data set, as well as variable metadata.\nUtilize the Harmony-py library to request subsetted MODIS L2 Sea Surface Temperature data over the Gulf of Mexico.\nRead Harmony subsetted outputs directly into xarray." + "objectID": "tutorials/04_On-Prem_Cloud.html#specify-data-time-range-and-area-of-interest", + "href": "tutorials/04_On-Prem_Cloud.html#specify-data-time-range-and-area-of-interest", + "title": "Pairing Cloud and non-Cloud Data", + "section": "Specify data, time range, and area of interest", + "text": "Specify data, time range, and area of interest\nWe are going to focus on getting data for an area north of Greenland for a single day in June.\nThese bounding_box and temporal variables will be used for data search, subset, and access below.\nThe same search and access steps for both datasets can be performed via Earthdata Search using the same spatial and temporal filtering options. See the Earthdata Search tutorial for more information on how to use Earthdata Search to discover and access data from the Earthdata Cloud.\n\n# Bounding Box spatial parameter in decimal degree 'W,S,E,N' format.\nbounding_box = '-62.8,81.7,-56.4,83'\n\n# Each date in yyyy-MM-ddTHH:mm:ssZ format; date range in start,end format\ntemporal = '2019-06-22T00:00:00Z,2019-06-22T23:59:59Z'\n\nSee the Data Discovery with CMR tutorial for more details on how to navigate the NASA Common Metadata Repository (CMR) Application Programming Interface, or API. For some background, the CMR catalogs all data for NASA’s Earth Observing System Data and Information System (EOSDIS). The CMR API allows applications (software, services, etc) to send information to each other. A helpful analogy is a waiter in a restaurant. The waiter takes your drink or food order that you select from the menu, often translated into short-hand, to the bar or kitchen, and then returns (hopefully) with what you ordered when it is ready.\nThe CMR API accepts search terms such as collection name, keywords, datetime range, and location, queries the CMR database and returns the results.\nFor this tutorial, we have already identified the unique identifier, or concept_id for each dataset:\n\nmodis_concept_id = 'C1940473819-POCLOUD'\nicesat2_concept_id = 'C2003771980-NSIDC_ECS'\n\nThis Earthdata Search Project also provides the same data access links that we will identify in the following steps for each dataset (note that you will need an Earthdata Login account to access this project)." }, { - "objectID": "examples/Transform-workflow.html#discover-service-options-for-a-given-data-set", - "href": "examples/Transform-workflow.html#discover-service-options-for-a-given-data-set", - "title": "Earthdata Cloud Exploration: Direct Access and Harmony workflow", - "section": "Discover service options for a given data set", - "text": "Discover service options for a given data set\n\nFirst, what do we mean by a “service”?\nIn the context of NASA Earthdata, we are usually referring to a service as any data transformation or customization process that packages or delivers data in a way that makes it easier to work with compared to how the data are natively archived at NASA EOSDIS. Basic customization options may include: * Subsetting (cropping) the data by: * Variable * Spatial boundary, * Temporal range * Reformatting * For example: From NetCDF-4 to Cloud Optimized GeoTIFF * Reprojection and/or Resampling * For example: From Sinusoidal to Polar Stereographic * Mosaicking * Aggregating\nA few main types or pathways for services that are commonly supported across the NASA DAACs include: * NASA Global Imagery Browse Service * Web services providing imagery, much of which is updated daily, to broaden accessibility of NASA EOSDIS data to the media and public. * OPeNDAP * The Open-source Project for a Network Data Access Protocol is a NASA community standard DAP that provides a simple way to access and work with data over the internet. OPeNDAP’s client/server software allows us to subset and reformat data using an internet browser, command line interface, and other applications. * Harmony * In the most basic sense, Harmony is an Application Programming Interface, or API, allowing us to request customization options described above, which are then processed and returned as file outputs. Harmony helps to reduce pre-processing steps so we can spend less time preparing the data, and more time doing science.\nNote: These service offerings are unique to each NASA EOSDIS dataset.\nWhy is this?\nDue to varying levels of service, cloud migration status, and unique characteristics of the datasets themselves, not all service options are provided for all datasets. Therefore it is important to first explore a given dataset’s metadata to discover what service options are provided.\n\n\nNote that the full Harmony tutorial from the 2021 Earthdata Cloud Hackathon demonstrates service and variable discovery, but this was removed here for simplification." + "objectID": "tutorials/04_On-Prem_Cloud.html#search-and-download-icesat-2-atl07-files", + "href": "tutorials/04_On-Prem_Cloud.html#search-and-download-icesat-2-atl07-files", + "title": "Pairing Cloud and non-Cloud Data", + "section": "Search and download ICESat-2 ATL07 files", + "text": "Search and download ICESat-2 ATL07 files\nPerform a granule search over our time and area of interest. How many granules are returned?\n\ngranule_url = 'https://cmr.earthdata.nasa.gov/search/granules'\n\n\nresponse = requests.get(granule_url,\n params={\n 'concept_id': icesat2_concept_id,\n 'temporal': temporal,\n 'bounding_box': bounding_box,\n 'page_size': 200,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\nprint(response.headers['CMR-Hits'])\n\n2\n\n\nPrint the file names, size, and links:\n\ngranules = response.json()['feed']['entry']\nfor granule in granules:\n print(f'{granule[\"producer_granule_id\"]} {granule[\"granule_size\"]} {granule[\"links\"][0][\"href\"]}')\n\nATL07-01_20190622055317_12980301_004_01.h5 237.0905504227 https://n5eil01u.ecs.nsidc.org/DP9/ATLAS/ATL07.004/2019.06.22/ATL07-01_20190622055317_12980301_004_01.h5\nATL07-01_20190622200154_13070301_004_01.h5 230.9151573181 https://n5eil01u.ecs.nsidc.org/DP9/ATLAS/ATL07.004/2019.06.22/ATL07-01_20190622200154_13070301_004_01.h5\n\n\n\nDownload ATL07 files\nAlthough several services are supported for ICESat-2 data, we are demonstrating direct access through the “on-prem” file system at NSIDC for simplicity.\nSome of these services include: - icepyx - From the icepyx documentation: “icepyx is both a software library and a community composed of ICESat-2 data users, developers, and the scientific community. We are working together to develop a shared library of resources - including existing resources, new code, tutorials, and use-cases/examples - that simplify the process of querying, obtaining, analyzing, and manipulating ICESat-2 datasets to enable scientific discovery.” - NSIDC DAAC Data Access and Service API - The API provided by the NSIDC DAAC allows you to access data programmatically using specific temporal and spatial filters. The same subsetting, reformatting, and reprojection services available on select data sets through NASA Earthdata Search can also be applied using this API. - IceFlow - The IceFlow python library simplifies accessing and combining data from several of NASA’s cryospheric altimetry missions, including ICESat/GLAS, Operation IceBridge, and ICESat-2. In particular, IceFlow harmonizes the various file formats and georeferencing parameters across several of the missions’ data sets, allowing you to analyze data across the multi-decadal time series.\nWe’ve found 2 granules. We’ll download the first one and write it to a file with the same name as the producer_granule_id.\nWe need the url for the granule as well. This is href links we printed out above.\n\nicesat_id = granules[0]['producer_granule_id']\nicesat_url = granules[0]['links'][0]['href']\n\nTo retrieve the granule data, we use the requests.get() method, which will utilize the .netrc file on the backend to authenticate the request against Earthdata Login.\n\nr = requests.get(icesat_url)\n\nThe response returned by requests has the same structure as all the other responses: a header and contents. The header information has information about the response, including the size of the data we downloaded in bytes.\n\nfor k, v in r.headers.items():\n print(f'{k}: {v}')\n\nDate: Sun, 12 Dec 2021 01:52:31 GMT\nServer: Apache\nVary: User-Agent\nContent-Disposition: attachment\nContent-Length: 248607461\nKeep-Alive: timeout=15, max=100\nConnection: Keep-Alive\n\n\nThe contents needs to be saved to a file. To keep the directory clean, we will create a downloads directory to store the file. We can use a shell command to do this or use the makedirs method from the os package.\n\nos.makedirs(\"downloads\", exist_ok=True)\n\nYou should see a downloads directory in the file browser.\nTo write the data to a file, we use open to open a file. We need to specify that the file is open for writing by using the write-mode w. We also need to specify that we want to write bytes by setting the binary-mode b. This is important because the response contents are bytes. The default mode for open is text-mode. So make sure you use b.\nWe’ll use the with statement context-manager to open the file, write the contents of the response, and then close the file. Once the data in r.content is written sucessfully to the file, or if there is an error, the file is closed by the context-manager.\nWe also need to prepend the downloads path to the filename. We do this using Path from the pathlib package in the standard library.\n\noutfile = Path('downloads', icesat_id)\n\n\nif not outfile.exists():\n with open(outfile, 'wb') as f:\n f.write(r.content)\n\nATL07-01_20190622055317_12980301_004_01.h5 is an HDF5 file. xarray can open this but you need to tell it which group to read the data from. In this case we read the sea ice segment height data for ground-track 1 left-beam. You can explore the variable hierarchy in Earthdata Search, by selecting the Customize option under Download Data.\nThis code block performs the following operations: - Extracts the height_segment_height variable from the heights group, along with the dimension variables contained in the higher level sea_ice_segments group, - Convert attributes from bytestrings to strings, - Drops the HDF attribute DIMENSION_LIST, - Sets _FillValue to NaN\n\nvariable_names = [\n '/gt1l/sea_ice_segments/latitude',\n '/gt1l/sea_ice_segments/longitude',\n '/gt1l/sea_ice_segments/delta_time',\n '/gt1l/sea_ice_segments/heights/height_segment_height'\n ]\nwith h5py.File(outfile, 'r') as h5:\n data_vars = {}\n for varname in variable_names:\n var = h5[varname]\n name = varname.split('/')[-1]\n # Convert attributes\n attrs = {}\n for k, v in var.attrs.items():\n if k != 'DIMENSION_LIST':\n if isinstance(v, bytes):\n attrs[k] = v.decode('utf-8')\n else:\n attrs[k] = v\n data = var[:]\n if '_FillValue' in attrs:\n data = np.where(data < attrs['_FillValue'], data, np.nan)\n data_vars[name] = (['segment'], data, attrs)\n is2_ds = xr.Dataset(data_vars)\n \nis2_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (segment: 235584)\nDimensions without coordinates: segment\nData variables:\n latitude (segment) float64 82.38 82.38 82.38 ... 72.61 72.61\n longitude (segment) float64 -55.11 -55.11 ... 145.1 145.1\n delta_time (segment) float64 4.642e+07 4.642e+07 ... 4.642e+07\n height_segment_height (segment) float32 nan nan nan ... -0.4335 -0.4463xarray.DatasetDimensions:segment: 235584Coordinates: (0)Data variables: (4)latitude(segment)float6482.38 82.38 82.38 ... 72.61 72.61contentType :referenceInformationcoordinates :delta_time longitudedescription :Latitude, WGS84, North=+, Lat of segment centerlong_name :Latitudesource :ATBD, section 4.4standard_name :latitudeunits :degrees_northvalid_max :90.0valid_min :-90.0array([82.38431982, 82.38431982, 82.38431982, ..., 72.60984638,\n 72.60977493, 72.60970985])longitude(segment)float64-55.11 -55.11 ... 145.1 145.1contentType :referenceInformationcoordinates :delta_time latitudedescription :Longitude, WGS84, East=+,Lon of segment centerlong_name :Longitudesource :ATBD, section 4.4standard_name :longitudeunits :degrees_eastvalid_max :180.0valid_min :-180.0array([-55.10896068, -55.10896068, -55.10896068, ..., 145.05396164,\n 145.05392851, 145.05389832])delta_time(segment)float644.642e+07 4.642e+07 ... 4.642e+07CLASS :DIMENSION_SCALENAME :gt1l/sea_ice_segments/delta_timeREFERENCE_LIST :[(<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)]contentType :physicalMeasurementcoordinates :latitude longitudedescription :Number of GPS seconds since the ATLAS SDP epoch. The ATLAS Standard Data Products (SDP) epoch offset is defined within /ancillary_data/atlas_sdp_gps_epoch as the number of GPS seconds between the GPS epoch (1980-01-06T00:00:00.000000Z UTC) and the ATLAS SDP epoch. By adding the offset contained within atlas_sdp_gps_epoch to delta time parameters, the time in gps_seconds relative to the GPS epoch can be computed.long_name :Elapsed GPS secondssource :telemetrystandard_name :timeunits :seconds since 2018-01-01array([46419293.64266939, 46419293.64266939, 46419293.64266939, ...,\n 46419681.87646231, 46419681.87759533, 46419681.87862704])height_segment_height(segment)float32nan nan nan ... -0.4335 -0.4463_FillValue :3.4028235e+38contentType :referenceInformationcoordinates :../delta_time ../latitude ../longitudedescription :Mean height from along-track segment fit detremined by the sea ice algorithm. The sea ice height is relative to the tide-free MSS.long_name :height of segment surfacesource :ATBD, section 4.2.2.4units :metersarray([ nan, nan, nan, ..., -0.46550068,\n -0.43347716, -0.4462675 ], dtype=float32)Attributes: (0)\n\n\n\nis2_ds.height_segment_height.plot() ;" }, { - "objectID": "examples/Transform-workflow.html#using-harmony-py-to-subset-data", - "href": "examples/Transform-workflow.html#using-harmony-py-to-subset-data", - "title": "Earthdata Cloud Exploration: Direct Access and Harmony workflow", - "section": "Using Harmony-Py to subset data", - "text": "Using Harmony-Py to subset data\nHarmony-Py provides a pip installable Python alternative to directly using Harmony’s RESTful API to make it easier to request data and service options, especially when interacting within a Python Jupyter Notebook environment.\nThe next steps are adopted from the introduction tutorial notebook provided in the Harmony-Py library:\n\nCreate Harmony Client object\nFirst, we need to create a Harmony Client, which is what we will interact with to submit and inspect a data request to Harmony, as well as to retrieve results.\nWhen creating the Client, we need to provide Earthdata Login credentials, which are required to access data from NASA EOSDIS. This basic line below assumes that we have a .netrc available.\n\nharmony_client = Client()\n\n\n\nCreate Harmony Request\nThe following are common request parameters:\n\ncollection: Required parameter. This is the NASA EOSDIS collection, or data product. There are two options for inputting a collection of interest:\n\nProvide a concept ID (e.g. C1940473819-POCLOUD)\nData product short name (e.g. MODIS_A-JPL-L2P-v2019.0).\n\nspatial: Bounding box spatial constraints on the data. The Harmony Bbox class accepts spatial coordinates as decimal degrees in w, s, e, n order, where longitude = -180, 180 and latitude = -90, 90.\ntemporal: Date/time constraints on the data. The example below demonstrates temporal start and end ranges using the python datetime library.\n\nAs we identified above, only subsetting options are available for this dataset. If other service options such as reformatting are available for a given dataset, these can also be specified using Harmony-py: See the documentation for details on how to construct these parameters.\n\nrequest = Request(\n collection=Collection(id=short_name),\n spatial=BBox(-97.77667,21.20806,-83.05197,30.16605),\n temporal={\n 'start': dt.datetime(2021, 8, 20),\n 'stop': dt.datetime(2021, 8, 21),\n },\n)\n\n\n\nSubmit request\nNow that the request is created, we can now submit it to Harmony using the Harmony Client object. A job id is returned, which is a unique identifier that represents the submitted request.\n\njob_id = harmony_client.submit(request)\njob_id\n\n'42f70e4f-5e6a-4473-8a5a-4ce56155ce92'\n\n\n\n\nCheck request status\nDepending on the size of the request, it may be helpful to wait until the request has completed processing before the remainder of the code is executed. The wait_for_processing() method will block subsequent lines of code while optionally showing a progress bar.\n\nharmony_client.wait_for_processing(job_id, show_progress=True)\n\n [ Processing: 100% ] |###################################################| [|]\n\n\n\n\nView Harmony job response and output URLs\nOnce the data request has finished processing, we can view details on the job that was submitted to Harmony, including the API call to Harmony, and informational messages on the request if available.\nresult_json() calls wait_for_processing() and returns the complete job in JSON format once processing is complete.\n\ndata = harmony_client.result_json(job_id)\npprint(data)\n\n{'createdAt': '2022-03-10T01:08:45.232Z',\n 'jobID': '42f70e4f-5e6a-4473-8a5a-4ce56155ce92',\n 'links': [{'href': 'https://harmony.earthdata.nasa.gov/stac/42f70e4f-5e6a-4473-8a5a-4ce56155ce92/',\n 'rel': 'stac-catalog-json',\n 'title': 'STAC catalog',\n 'type': 'application/json'},\n {'bbox': [-97.8, 21.2, -90.8, 22.5],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820172501-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T17:29:59.000Z',\n 'start': '2021-08-20T17:25:01.000Z'},\n 'title': '20210820172501-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-97.8, 21.2, -86.8, 30.2],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820172001-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T17:24:59.000Z',\n 'start': '2021-08-20T17:20:01.000Z'},\n 'title': '20210820172001-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-88.2, 21.2, -83.1, 28.3],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820154501-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T15:49:59.000Z',\n 'start': '2021-08-20T15:45:01.000Z'},\n 'title': '20210820154501-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-87.1, 28, -83.1, 30.2],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820154001-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T15:44:59.000Z',\n 'start': '2021-08-20T15:40:01.000Z'},\n 'title': '20210820154001-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-97.8, 21.2, -94, 30.2],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820051501-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T05:19:59.000Z',\n 'start': '2021-08-20T05:15:01.000Z'},\n 'title': '20210820051501-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-95.4, 21.2, -83.1, 30.2],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820033500-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T03:39:59.000Z',\n 'start': '2021-08-20T03:35:00.000Z'},\n 'title': '20210820033500-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'href': 'https://harmony.earthdata.nasa.gov/jobs/42f70e4f-5e6a-4473-8a5a-4ce56155ce92?linktype=https&page=1&limit=2000',\n 'rel': 'self',\n 'title': 'The current page',\n 'type': 'application/json'}],\n 'message': 'The job has completed successfully',\n 'numInputGranules': 6,\n 'progress': 100,\n 'request': 'https://harmony.earthdata.nasa.gov/MODIS_T-JPL-L2P-v2019.0/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(21.20806%3A30.16605)&subset=lon(-97.77667%3A-83.05197)&subset=time(%222021-08-20T00%3A00%3A00%22%3A%222021-08-21T00%3A00%3A00%22)',\n 'status': 'successful',\n 'updatedAt': '2022-03-10T01:09:26.492Z',\n 'username': 'amy.steiker'}\n\n\n\n\nDirect cloud access\nNote that the remainder of this tutorial will only succeed when running this notebook within the AWS us-west-2 region.\nHarmony data outputs can be accessed within the cloud using the s3 URLs and AWS credentials provided in the Harmony job response.\n\nRetrieve list of output URLs.\nThe result_urls() method calls wait_for_processing() and returns a list of the processed data URLs once processing is complete. You may optionally show the progress bar as shown below.\n\nresults = harmony_client.result_urls(job_id, link_type=LinkType.s3)\nurls = list(results)\npprint(urls)\n\n['s3://harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820172501-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820172001-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820154501-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820154001-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820051501-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0_subsetted.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820033500-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0_subsetted.nc4']\n\n\nWe’ll select a URL in the list to bring into xarray below, which matches the granule we opened directly from PO.DAAC’s archive bucket above. In Earthdata Search, we can see that this granule covers our area well.\n\nfilename = '20210820033500-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0_subsetted.nc4'\n\nurl = [url for url in urls if filename in url][0]\nurl\n\n's3://harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820033500-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0_subsetted.nc4'\n\n\n\n\nAWS credential retrieval\nUsing aws_credentials you can retrieve the credentials needed to access the Harmony s3 staging bucket and its contents.\n\ncreds = harmony_client.aws_credentials()\n\n\n\n\nOpen staged files with s3fs and xarray\nWe use the AWS s3fs package to create a file system that can then be read by xarray:\n\ns3_fs = s3fs.S3FileSystem(\n key=creds['aws_access_key_id'],\n secret=creds['aws_secret_access_key'],\n token=creds['aws_session_token'],\n client_kwargs={'region_name':'us-west-2'},\n)\n\nNow that we have our s3 file system set, including our declared credentials, we’ll use that to open the url, and read in the file through xarray. This extra step is needed because xarray cannot open the S3 location directly. Instead, the S3 file object is passed to xarray, in order to then open the dataset.\n\nf = s3_fs.open(url, mode='rb')\nds = xr.open_dataset(f)\nds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (nj: 1228, ni: 660, time: 1)\nCoordinates:\n lat (nj, ni, time) float32 ...\n lon (nj, ni, time) float32 ...\n * time (time) datetime64[ns] 2021-08-20T03:35:00\nDimensions without coordinates: nj, ni\nData variables:\n sea_surface_temperature (time, nj, ni) float32 ...\n sst_dtime (time, nj, ni) timedelta64[ns] ...\n quality_level (time, nj, ni) float32 ...\n sses_bias (time, nj, ni) float32 ...\n sses_standard_deviation (time, nj, ni) float32 ...\n l2p_flags (time, nj, ni) int16 ...\n sea_surface_temperature_4um (time, nj, ni) float32 ...\n quality_level_4um (time, nj, ni) float32 ...\n sses_bias_4um (time, nj, ni) float32 ...\n sses_standard_deviation_4um (time, nj, ni) float32 ...\n wind_speed (time, nj, ni) float32 ...\n dt_analysis (time, nj, ni) float32 ...\nAttributes: (12/50)\n Conventions: CF-1.7, ACDD-1.3\n title: MODIS Terra L2P SST\n summary: Sea surface temperature retrievals produced a...\n references: GHRSST Data Processing Specification v2r5\n institution: NASA/JPL/OBPG/RSMAS\n history: MODIS L2P created at JPL PO.DAAC\\n2022-03-10 ...\n ... ...\n processing_level: L2P\n cdm_data_type: swath\n startDirection: Ascending\n endDirection: Ascending\n day_night_flag: Night\n history_json: [{\"date_time\": \"2022-03-10T01:09:25.100329+00...xarray.DatasetDimensions:nj: 1228ni: 660time: 1Coordinates: (3)lat(nj, ni, time)float32...long_name :latitudestandard_name :latitudeunits :degrees_northvalid_min :-90.0valid_max :90.0comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[810480 values with dtype=float32]lon(nj, ni, time)float32...long_name :longitudestandard_name :longitudeunits :degrees_eastvalid_min :-180.0valid_max :180.0comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[810480 values with dtype=float32]time(time)datetime64[ns]2021-08-20T03:35:00long_name :reference time of sst filestandard_name :timecomment :time of first sensor observationcoverage_content_type :coordinatearray(['2021-08-20T03:35:00.000000000'], dtype='datetime64[ns]')Data variables: (12)sea_surface_temperature(time, nj, ni)float32...long_name :sea surface temperaturestandard_name :sea_surface_skin_temperatureunits :kelvinvalid_min :-1000valid_max :10000comment :sea surface temperature from thermal IR (11 um) channelssource :NASA and University of Miamicoverage_content_type :physicalMeasurement[810480 values with dtype=float32]sst_dtime(time, nj, ni)timedelta64[ns]...long_name :time difference from reference timevalid_min :-32767valid_max :32767comment :time plus sst_dtime gives seconds after 00:00:00 UTC January 1, 1981coverage_content_type :referenceInformation[810480 values with dtype=timedelta64[ns]]quality_level(time, nj, ni)float32...long_name :quality level of SST pixelvalid_min :0valid_max :5comment :thermal IR SST proximity confidence value; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valueflag_values :[0 1 2 3 4 5]flag_meanings :no_data bad_data worst_quality low_quality acceptable_quality best_qualitycoverage_content_type :qualityInformation[810480 values with dtype=float32]sses_bias(time, nj, ni)float32...long_name :SSES bias error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :thermal IR SST bias error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[810480 values with dtype=float32]sses_standard_deviation(time, nj, ni)float32...long_name :SSES standard deviation error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :thermal IR SST standard deviation error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[810480 values with dtype=float32]l2p_flags(time, nj, ni)int16...long_name :L2P flagsvalid_min :0valid_max :16comment :These flags can be used to further filter data variablesflag_meanings :microwave land ice lake riverflag_masks :[ 1 2 4 8 16]coverage_content_type :qualityInformation[810480 values with dtype=int16]sea_surface_temperature_4um(time, nj, ni)float32...long_name :sea surface temperatureunits :kelvinvalid_min :-1000valid_max :10000comment :sea surface temperature from mid-IR (4 um) channels; non L2P core fieldcoverage_content_type :physicalMeasurement[810480 values with dtype=float32]quality_level_4um(time, nj, ni)float32...long_name :quality level of SST pixelvalid_min :0valid_max :5comment :mid-IR SST proximity confidence value; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valueflag_values :[0 1 2 3 4 5]flag_meanings :no_data bad_data worst_quality low_quality acceptable_quality best_qualitycoverage_content_type :qualityInformation[810480 values with dtype=float32]sses_bias_4um(time, nj, ni)float32...long_name :SSES bias error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :mid-IR SST bias error; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[810480 values with dtype=float32]sses_standard_deviation_4um(time, nj, ni)float32...long_name :SSES standard deviation error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :mid-IR SST standard deviation error; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[810480 values with dtype=float32]wind_speed(time, nj, ni)float32...long_name :10m wind speedstandard_name :wind_speedunits :m s-1valid_min :-127valid_max :127comment :Wind at 10 meters above the sea surfacesource :TBD. Placeholder. Currently emptygrid_mapping :TBDtime_offset :2.0height :10 mcoverage_content_type :auxiliaryInformation[810480 values with dtype=float32]dt_analysis(time, nj, ni)float32...long_name :deviation from SST reference climatologyunits :kelvinvalid_min :-127valid_max :127comment :TBDsource :TBD. Placeholder. Currently emptycoverage_content_type :auxiliaryInformation[810480 values with dtype=float32]Attributes: (50)Conventions :CF-1.7, ACDD-1.3title :MODIS Terra L2P SSTsummary :Sea surface temperature retrievals produced at the NASA OBPG for the MODIS Aqua sensor. These have been reformatted to GHRSST GDS specifications by the JPL PO.DAACreferences :GHRSST Data Processing Specification v2r5institution :NASA/JPL/OBPG/RSMAShistory :MODIS L2P created at JPL PO.DAAC\n2022-03-10 01:09:25.100285 l2ss-py v1.3.1 (bbox=[[-97.77667, -83.05197], [21.20806, 30.16605]] cut=True)comment :L2P Core without DT analysis or other ancillary fields; Night, Start Node:Ascending, End Node:Ascending; WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value; Quicklooklicense :GHRSST and PO.DAAC protocol allow data use as free and open.id :MODIS_T-JPL-L2P-v2019.0naming_authority :org.ghrsstproduct_version :2019.0uuid :f6e1f61d-c4a4-4c17-8354-0c15e12d688bgds_version_id :2.0netcdf_version_id :4.1date_created :20210820T153042Zfile_quality_level :3spatial_resolution :1kmstart_time :20210820T033500Ztime_coverage_start :20210820T033500Zstop_time :20210820T033959Ztime_coverage_end :20210820T033959Znorthernmost_latitude :34.0296southernmost_latitude :12.9895easternmost_longitude :-68.4372westernmost_longitude :-95.5518source :MODIS sea surface temperature observations for the OBPGplatform :Terrasensor :MODISmetadata_link :http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MODIS_T-JPL-L2P-v2019.0keywords :Oceans > Ocean Temperature > Sea Surface Temperaturekeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordsstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventiongeospatial_lat_units :degrees_northgeospatial_lat_resolution :0.01geospatial_lon_units :degrees_eastgeospatial_lon_resolution :0.01acknowledgment :The MODIS L2P sea surface temperature data are sponsored by NASAcreator_name :Ed Armstrong, JPL PO.DAACcreator_email :edward.m.armstrong@jpl.nasa.govcreator_url :http://podaac.jpl.nasa.govproject :Group for High Resolution Sea Surface Temperaturepublisher_name :The GHRSST Project Officepublisher_url :http://www.ghrsst.orgpublisher_email :ghrsst-po@nceo.ac.ukprocessing_level :L2Pcdm_data_type :swathstartDirection :AscendingendDirection :Ascendingday_night_flag :Nighthistory_json :[{\"date_time\": \"2022-03-10T01:09:25.100329+00:00\", \"derived_from\": \"https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MODIS_T-JPL-L2P-v2019.0/20210820033500-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0.nc\", \"program\": \"l2ss-py\", \"version\": \"1.3.1\", \"parameters\": \"bbox=[[-97.77667, -83.05197], [21.20806, 30.16605]] cut=True\", \"program_ref\": \"https://cmr.earthdata.nasa.gov:443/search/concepts/S1962070864-POCLOUD\", \"$schema\": \"https://harmony.earthdata.nasa.gov/schemas/history/0.1.0/history-v0.1.0.json\"}]\n\n\n\n\nPlot the data\nUse the xarray built in plotting function to create a simple plot along the x and y dimensions of the dataset:\n\nds.sea_surface_temperature.plot() ;\n\n\n\n\n\n\nSummary\nThis notebook demonstrated an abbreviated and simplified workflow to explore access and subsetting options available through the Earthdata Cloud. There are several other options that can be used to work “in place” in the Earthdata Cloud, from data discovery to analysis-ready data, including:\n\nZarr-EOSDIS-Store\n\nThe zarr-eosdis-store library allows NASA EOSDIS Collections to be accessed efficiently by the Zarr Python library, provided they have a sidecar DMR++ metadata file generated.\nTutorial highlighting this library’s usage: https://nasa-openscapes.github.io/2021-Cloud-Hackathon/tutorials/09_Zarr_Access.html\n\nOPeNDAP\n\nHyrax provides direct access to subsetting of NASA data using Python or your favorite analysis tool\nTutorial highlighting OPeNDAP usage: https://nasa-openscapes.github.io/2021-Cloud-Hackathon/tutorials/06_S6_OPeNDAP_Access_Gridding.html\n\nEarthdata Library\n\nA Python library to search and access NASA Earthdata datasets, aiming to provide a simple way to access or download data without having to worry if a given dataset is on-prem or in the cloud." + "objectID": "tutorials/04_On-Prem_Cloud.html#determine-the-ghrsst-modis-l2-granules-returned-from-our-time-and-area-of-interest", + "href": "tutorials/04_On-Prem_Cloud.html#determine-the-ghrsst-modis-l2-granules-returned-from-our-time-and-area-of-interest", + "title": "Pairing Cloud and non-Cloud Data", + "section": "Determine the GHRSST MODIS L2 granules returned from our time and area of interest", + "text": "Determine the GHRSST MODIS L2 granules returned from our time and area of interest\n\nresponse = requests.get(granule_url, \n params={\n 'concept_id': modis_concept_id,\n 'temporal': temporal,\n 'bounding_box': bounding_box,\n 'page_size': 200,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\nprint(response.headers['CMR-Hits'])\n\n14\n\n\n\ngranules = response.json()['feed']['entry']\nfor granule in granules:\n print(f'{granule[\"title\"]} {granule[\"granule_size\"]} {granule[\"links\"][0][\"href\"]}')\n\n20190622000501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 20.71552562713623 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622000501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622014501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 9.34600830078125E-5 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622014501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622032501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 21.307741165161133 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622032501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622050001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 9.34600830078125E-5 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622050001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622050501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 20.065649032592773 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622050501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622064001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 9.34600830078125E-5 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622064001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622064501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0 18.602201461791992 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622064501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\n20190622064501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 18.665077209472656 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622064501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622082001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 19.782299995422363 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622082001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622100001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 19.13440227508545 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622100001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622113501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 20.3239164352417 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622113501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622114001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 9.34600830078125E-5 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622114001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622163001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 19.257243156433105 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622163001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622181001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 19.93498420715332 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622181001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc" }, { - "objectID": "examples/ORNL/ORNL_DAYMET__Kerchunk.html", - "href": "examples/ORNL/ORNL_DAYMET__Kerchunk.html", - "title": "EarthData Cloud Cookbook", - "section": "", - "text": "!mamba install kerchunk -y --quiet\n\n\n!mamba install h5py=3.2 -y --quiet\n\n\nimport numpy as np\nimport xarray as xr\nimport matplotlib.pyplot as plt\nimport s3fs\nimport datetime as dt\nimport logging\nimport fsspec\nimport ujson\nimport requests\nfrom tqdm import tqdm\nfrom glob import glob\n\n\nimport kerchunk.hdf\n\n\ns3_cred_endpoint = {\n 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n 'ornldaac':'https://data.ornldaac.earthdata.nasa.gov/s3credentials'\n}\n\n\ndef get_temp_creds():\n temp_creds_url = s3_cred_endpoint['ornldaac']\n return requests.get(temp_creds_url).json()\n\n\ntemp_creds_req = get_temp_creds()\n\n\n#fs = fsspec.filesystem('s3', anon=True)\nfs = s3fs.S3FileSystem(anon=False, key=temp_creds_req['accessKeyId'], secret=temp_creds_req['secretAccessKey'], token=temp_creds_req['sessionToken'])\n\n\nurls = ['s3://ornl-cumulus-prod-protected/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_tmax_2015.nc',\n 's3://ornl-cumulus-prod-protected/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_tmax_2016.nc',\n 's3://ornl-cumulus-prod-protected/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_tmax_2017.nc',\n 's3://ornl-cumulus-prod-protected/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_tmax_2018.nc',\n 's3://ornl-cumulus-prod-protected/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_tmax_2019.nc',\n 's3://ornl-cumulus-prod-protected/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_tmax_2020.nc']\n\n\ndef gen_json(u):\n so = dict(\n mode=\"rb\", anon=False, default_fill_cache=False,\n default_cache_type=\"none\"\n )\n with fs.open(u, **so) as infile:\n print(u)\n h5chunks = kerchunk.hdf.SingleHdf5ToZarr(infile, u, inline_threshold=300)\n with open(f\"jsons/{u.split('/')[-1]}.json\", 'wb') as outfile:\n outfile.write(ujson.dumps(h5chunks.translate()).encode())\n\n\nfor u in urls:\n gen_json(u)\n\n\nu = 's3://ornl-cumulus-prod-protected/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_tmax_2015.nc'\n\n\nso = dict(\n mode=\"rb\", anon=False, default_fill_cache=False, default_cache_type=\"none\"\n )\n\n\ninf = fs.open(u, **so)\n\n\nh5chunks = kerchunk.hdf.SingleHdf5ToZarr(inf, u, inline_threshold=300)\n\n\nh5chunks.translate()\n\nPermissionError: The provided token has expired." + "objectID": "tutorials/04_On-Prem_Cloud.html#load-data-into-xarray-via-s3-direct-access", + "href": "tutorials/04_On-Prem_Cloud.html#load-data-into-xarray-via-s3-direct-access", + "title": "Pairing Cloud and non-Cloud Data", + "section": "Load data into xarray via S3 direct access", + "text": "Load data into xarray via S3 direct access\nOur CMR granule search returned 14 files for our time and area of interest. However, not all granules will be suitable for analysis.\nI’ve identified the image with granule id G1956158784-POCLOUD as a good candidate, this is the 9th granule. In this image, our area of interest is close to nadir. This means that the instantaneous field of view over the area of interest cover a smaller area than at the edge of the image.\nWe are looking for the link for direct download access via s3. This is a url but with a prefix s3://. This happens to be the first href link in the metadata.\nFor a single granule we can cut and paste the s3 link. If we have several granules, the s3 links can be extracted with some simple code.\n\ngranule = granules[9]\n\nfor link in granule['links']:\n if link['href'].startswith('s3://'):\n s3_link = link['href']\n \ns3_link\n\n's3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622100001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc'\n\n\n\nGet S3 credentials\nAs with the previous S3 download tutorials we need credentials to access data from s3: access keys and tokens.\n\ns3_credentials = requests.get('https://archive.podaac.earthdata.nasa.gov/s3credentials').json()\n\nEssentially, what we are doing in this step is to “mount” the s3 bucket as a file system. This allows us to treat the S3 bucket in a similar way to a local file system.\n\ns3_fs = s3fs.S3FileSystem(\n key=s3_credentials[\"accessKeyId\"],\n secret=s3_credentials[\"secretAccessKey\"],\n token=s3_credentials[\"sessionToken\"],\n)\n\n\n\nOpen a s3 file\nNow we have the S3FileSystem set up, we can access the granule. xarray cannot open a S3File directly, so we use the open method for the S3FileSystem to open the granule using the endpoint url we extracted from the metadata. We also have to set the mode='rb'. This opens the granule in read-only mode and in byte-mode. Byte-mode is important. By default, open opens a file as text - in this case it would just be a string of characters - and xarray doesn’t know what to do with that.\nWe then pass the S3File object f to xarray.open_dataset. For this dataset, we also have to set decode_cf=False. This switch tells xarray not to use information contained in variable attributes to generate human readable coordinate variables. Normally, this should work for netcdf files but for this particular cloud-hosted dataset, variable attribute data is not in the form expected by xarray. We’ll fix this.\n\nf = s3_fs.open(s3_link, mode='rb')\nmodis_ds = xr.open_dataset(f, decode_cf=False)\n\nIf you click on the Show/Hide Attributes icon (the first document-like icon to the right of coordinate variable metadata) you can see that attributes are one-element arrays containing bytestrings.\n\nmodis_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (nj: 2030, ni: 1354, time: 1)\nCoordinates:\n * time (time) int32 1214042401\nDimensions without coordinates: nj, ni\nData variables:\n lat (nj, ni) float32 ...\n lon (nj, ni) float32 ...\n sea_surface_temperature (time, nj, ni) int16 ...\n sst_dtime (time, nj, ni) int16 ...\n quality_level (time, nj, ni) int8 ...\n sses_bias (time, nj, ni) int8 ...\n sses_standard_deviation (time, nj, ni) int8 ...\n l2p_flags (time, nj, ni) int16 ...\n chlorophyll_a (time, nj, ni) float32 ...\n K_490 (time, nj, ni) int16 ...\n wind_speed (time, nj, ni) int8 ...\n dt_analysis (time, nj, ni) int8 ...\nAttributes: (12/49)\n Conventions: [b'CF-1.7, ACDD-1.3']\n title: [b'MODIS Aqua L2P SST']\n summary: [b'Sea surface temperature retrievals produce...\n references: [b'GHRSST Data Processing Specification v2r5']\n institution: [b'NASA/JPL/OBPG/RSMAS']\n history: [b'MODIS L2P created at JPL PO.DAAC']\n ... ...\n publisher_email: [b'ghrsst-po@nceo.ac.uk']\n processing_level: [b'L2P']\n cdm_data_type: [b'swath']\n startDirection: [b'Ascending']\n endDirection: [b'Descending']\n day_night_flag: [b'Day']xarray.DatasetDimensions:nj: 2030ni: 1354time: 1Coordinates: (1)time(time)int321214042401long_name :[b'reference time of sst file']standard_name :[b'time']units :[b'seconds since 1981-01-01 00:00:00']comment :[b'time of first sensor observation']coverage_content_type :[b'coordinate']array([1214042401], dtype=int32)Data variables: (12)lat(nj, ni)float32...long_name :[b'latitude']standard_name :[b'latitude']units :[b'degrees_north']_FillValue :[-999.]valid_min :[-90.]valid_max :[90.]comment :[b'geographical coordinates, WGS84 projection']coverage_content_type :[b'coordinate'][2748620 values with dtype=float32]lon(nj, ni)float32...long_name :[b'longitude']standard_name :[b'longitude']units :[b'degrees_east']_FillValue :[-999.]valid_min :[-180.]valid_max :[180.]comment :[b'geographical coordinates, WGS84 projection']coverage_content_type :[b'coordinate'][2748620 values with dtype=float32]sea_surface_temperature(time, nj, ni)int16...long_name :[b'sea surface temperature']standard_name :[b'sea_surface_skin_temperature']units :[b'kelvin']_FillValue :[-32767]valid_min :[-1000]valid_max :[10000]comment :[b'sea surface temperature from thermal IR (11 um) channels']scale_factor :[0.005]add_offset :[273.15]source :[b'NASA and University of Miami']coordinates :[b'lon lat']coverage_content_type :[b'physicalMeasurement'][2748620 values with dtype=int16]sst_dtime(time, nj, ni)int16...long_name :[b'time difference from reference time']units :[b'seconds']_FillValue :[-32768]valid_min :[-32767]valid_max :[32767]comment :[b'time plus sst_dtime gives seconds after 00:00:00 UTC January 1, 1981']coordinates :[b'lon lat']coverage_content_type :[b'referenceInformation'][2748620 values with dtype=int16]quality_level(time, nj, ni)int8...long_name :[b'quality level of SST pixel']_FillValue :[-128]valid_min :[0]valid_max :[5]comment :[b'thermal IR SST proximity confidence value; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value']coordinates :[b'lon lat']flag_values :[0 1 2 3 4 5]flag_meanings :[b'no_data bad_data worst_quality low_quality acceptable_quality best_quality']coverage_content_type :[b'qualityInformation'][2748620 values with dtype=int8]sses_bias(time, nj, ni)int8...long_name :[b'SSES bias error based on proximity confidence flags']units :[b'kelvin']_FillValue :[-128]valid_min :[-127]valid_max :[127]comment :[b'thermal IR SST bias error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value']scale_factor :[0.15748031]add_offset :[0.]coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int8]sses_standard_deviation(time, nj, ni)int8...long_name :[b'SSES standard deviation error based on proximity confidence flags']units :[b'kelvin']_FillValue :[-128]valid_min :[-127]valid_max :[127]comment :[b'thermal IR SST standard deviation error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value']scale_factor :[0.07874016]add_offset :[10.]coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int8]l2p_flags(time, nj, ni)int16...long_name :[b'L2P flags']valid_min :[0]valid_max :[16]comment :[b'These flags can be used to further filter data variables']coordinates :[b'lon lat']flag_meanings :[b'microwave land ice lake river']flag_masks :[ 1 2 4 8 16]coverage_content_type :[b'qualityInformation'][2748620 values with dtype=int16]chlorophyll_a(time, nj, ni)float32...long_name :[b'Chlorophyll Concentration, OC3 Algorithm']units :[b'mg m^-3']_FillValue :[-32767.]valid_min :[0.001]valid_max :[100.]comment :[b'non L2P core field']coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=float32]K_490(time, nj, ni)int16...long_name :[b'Diffuse attenuation coefficient at 490 nm (OBPG)']units :[b'm^-1']_FillValue :[-32767]valid_min :[50]valid_max :[30000]comment :[b'non L2P core field']scale_factor :[0.0002]add_offset :[0.]coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int16]wind_speed(time, nj, ni)int8...long_name :[b'10m wind speed']standard_name :[b'wind_speed']units :[b'm s-1']_FillValue :[-128]valid_min :[-127]valid_max :[127]comment :[b'Wind at 10 meters above the sea surface']scale_factor :[0.2]add_offset :[25.]source :[b'TBD. Placeholder. Currently empty']coordinates :[b'lon lat']grid_mapping :[b'TBD']time_offset :[2.]height :[b'10 m']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int8]dt_analysis(time, nj, ni)int8...long_name :[b'deviation from SST reference climatology']units :[b'kelvin']_FillValue :[-128]valid_min :[-127]valid_max :[127]comment :[b'TBD']scale_factor :[0.1]add_offset :[0.]source :[b'TBD. Placeholder. Currently empty']coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int8]Attributes: (49)Conventions :[b'CF-1.7, ACDD-1.3']title :[b'MODIS Aqua L2P SST']summary :[b'Sea surface temperature retrievals produced at the NASA OBPG for the MODIS Aqua sensor. These have been reformatted to GHRSST GDS specifications by the JPL PO.DAAC']references :[b'GHRSST Data Processing Specification v2r5']institution :[b'NASA/JPL/OBPG/RSMAS']history :[b'MODIS L2P created at JPL PO.DAAC']comment :[b'L2P Core without DT analysis or other ancillary fields; Day, Start Node:Ascending, End Node:Descending; WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value; Refined']license :[b'GHRSST and PO.DAAC protocol allow data use as free and open.']id :[b'MODIS_A-JPL-L2P-v2019.0']naming_authority :[b'org.ghrsst']product_version :[b'2019.0']uuid :[b'f6e1f61d-c4a4-4c17-8354-0c15e12d688b']gds_version_id :[b'2.0']netcdf_version_id :[b'4.1']date_created :[b'20200221T085224Z']file_quality_level :[3]spatial_resolution :[b'1km']start_time :[b'20190622T100001Z']time_coverage_start :[b'20190622T100001Z']stop_time :[b'20190622T100459Z']time_coverage_end :[b'20190622T100459Z']northernmost_latitude :[89.9862]southernmost_latitude :[66.2723]easternmost_longitude :[-45.9467]westernmost_longitude :[152.489]source :[b'MODIS sea surface temperature observations for the OBPG']platform :[b'Aqua']sensor :[b'MODIS']metadata_link :[b'http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MODIS_A-JPL-L2P-v2019.0']keywords :[b'Oceans > Ocean Temperature > Sea Surface Temperature']keywords_vocabulary :[b'NASA Global Change Master Directory (GCMD) Science Keywords']standard_name_vocabulary :[b'NetCDF Climate and Forecast (CF) Metadata Convention']geospatial_lat_units :[b'degrees_north']geospatial_lat_resolution :[0.01]geospatial_lon_units :[b'degrees_east']geospatial_lon_resolution :[0.01]acknowledgment :[b'The MODIS L2P sea surface temperature data are sponsored by NASA']creator_name :[b'Ed Armstrong, JPL PO.DAAC']creator_email :[b'edward.m.armstrong@jpl.nasa.gov']creator_url :[b'http://podaac.jpl.nasa.gov']project :[b'Group for High Resolution Sea Surface Temperature']publisher_name :[b'The GHRSST Project Office']publisher_url :[b'http://www.ghrsst.org']publisher_email :[b'ghrsst-po@nceo.ac.uk']processing_level :[b'L2P']cdm_data_type :[b'swath']startDirection :[b'Ascending']endDirection :[b'Descending']day_night_flag :[b'Day']\n\n\nTo fix this, we need to extract array elements as scalars, and convert those scalars from bytestrings to strings. We use the decode method to do this. The bytestrings are encoded as utf-8, which is a unicode character format. This is the default encoding for decode but we’ve included it as an argument to be explicit.\nNot all attributes are bytestrings. Some are floats. Take a look at _FillValue, and valid_min and valid_max. To avoid an error, we use the isinstance function to check if the value of an attributes is type bytes - a bytestring. If it is, then we decode it. If not, we just extract the scalar and do nothing else.\nWe also fix the global attributes.\n\ndef fix_attributes(da):\n '''Decodes bytestring attributes to strings'''\n for attr, value in da.attrs.items():\n if isinstance(value[0], bytes):\n da.attrs[attr] = value[0].decode('utf-8')\n else:\n da.attrs[attr] = value[0]\n return\n\n# Fix variable attributes\nfor var in modis_ds.variables:\n da = modis_ds[var]\n fix_attributes(da)\n \n# Fix global attributes\nfix_attributes(modis_ds)\n\nWith this done, we can use the xarray function decode_cf to convert the attributes.\n\nmodis_ds = xr.decode_cf(modis_ds)\n\n\nmodis_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (nj: 2030, ni: 1354, time: 1)\nCoordinates:\n lat (nj, ni) float32 ...\n lon (nj, ni) float32 ...\n * time (time) datetime64[ns] 2019-06-22T10:00:01\nDimensions without coordinates: nj, ni\nData variables:\n sea_surface_temperature (time, nj, ni) float32 ...\n sst_dtime (time, nj, ni) timedelta64[ns] ...\n quality_level (time, nj, ni) float32 ...\n sses_bias (time, nj, ni) float32 ...\n sses_standard_deviation (time, nj, ni) float32 ...\n l2p_flags (time, nj, ni) int16 ...\n chlorophyll_a (time, nj, ni) float32 ...\n K_490 (time, nj, ni) float32 ...\n wind_speed (time, nj, ni) float32 ...\n dt_analysis (time, nj, ni) float32 ...\nAttributes: (12/49)\n Conventions: CF-1.7, ACDD-1.3\n title: MODIS Aqua L2P SST\n summary: Sea surface temperature retrievals produced a...\n references: GHRSST Data Processing Specification v2r5\n institution: NASA/JPL/OBPG/RSMAS\n history: MODIS L2P created at JPL PO.DAAC\n ... ...\n publisher_email: ghrsst-po@nceo.ac.uk\n processing_level: L2P\n cdm_data_type: swath\n startDirection: Ascending\n endDirection: Descending\n day_night_flag: Dayxarray.DatasetDimensions:nj: 2030ni: 1354time: 1Coordinates: (3)lat(nj, ni)float32...long_name :latitudestandard_name :latitudeunits :degrees_northvalid_min :-90.0valid_max :90.0comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[2748620 values with dtype=float32]lon(nj, ni)float32...long_name :longitudestandard_name :longitudeunits :degrees_eastvalid_min :-180.0valid_max :180.0comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[2748620 values with dtype=float32]time(time)datetime64[ns]2019-06-22T10:00:01long_name :reference time of sst filestandard_name :timecomment :time of first sensor observationcoverage_content_type :coordinatearray(['2019-06-22T10:00:01.000000000'], dtype='datetime64[ns]')Data variables: (10)sea_surface_temperature(time, nj, ni)float32...long_name :sea surface temperaturestandard_name :sea_surface_skin_temperatureunits :kelvinvalid_min :-1000valid_max :10000comment :sea surface temperature from thermal IR (11 um) channelssource :NASA and University of Miamicoverage_content_type :physicalMeasurement[2748620 values with dtype=float32]sst_dtime(time, nj, ni)timedelta64[ns]...long_name :time difference from reference timevalid_min :-32767valid_max :32767comment :time plus sst_dtime gives seconds after 00:00:00 UTC January 1, 1981coverage_content_type :referenceInformation[2748620 values with dtype=timedelta64[ns]]quality_level(time, nj, ni)float32...long_name :quality level of SST pixelvalid_min :0valid_max :5comment :thermal IR SST proximity confidence value; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valueflag_values :0flag_meanings :no_data bad_data worst_quality low_quality acceptable_quality best_qualitycoverage_content_type :qualityInformation[2748620 values with dtype=float32]sses_bias(time, nj, ni)float32...long_name :SSES bias error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :thermal IR SST bias error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]sses_standard_deviation(time, nj, ni)float32...long_name :SSES standard deviation error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :thermal IR SST standard deviation error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]l2p_flags(time, nj, ni)int16...long_name :L2P flagsvalid_min :0valid_max :16comment :These flags can be used to further filter data variablesflag_meanings :microwave land ice lake riverflag_masks :1coverage_content_type :qualityInformation[2748620 values with dtype=int16]chlorophyll_a(time, nj, ni)float32...long_name :Chlorophyll Concentration, OC3 Algorithmunits :mg m^-3valid_min :0.001valid_max :100.0comment :non L2P core fieldcoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]K_490(time, nj, ni)float32...long_name :Diffuse attenuation coefficient at 490 nm (OBPG)units :m^-1valid_min :50valid_max :30000comment :non L2P core fieldcoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]wind_speed(time, nj, ni)float32...long_name :10m wind speedstandard_name :wind_speedunits :m s-1valid_min :-127valid_max :127comment :Wind at 10 meters above the sea surfacesource :TBD. Placeholder. Currently emptygrid_mapping :TBDtime_offset :2.0height :10 mcoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]dt_analysis(time, nj, ni)float32...long_name :deviation from SST reference climatologyunits :kelvinvalid_min :-127valid_max :127comment :TBDsource :TBD. Placeholder. Currently emptycoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]Attributes: (49)Conventions :CF-1.7, ACDD-1.3title :MODIS Aqua L2P SSTsummary :Sea surface temperature retrievals produced at the NASA OBPG for the MODIS Aqua sensor. These have been reformatted to GHRSST GDS specifications by the JPL PO.DAACreferences :GHRSST Data Processing Specification v2r5institution :NASA/JPL/OBPG/RSMAShistory :MODIS L2P created at JPL PO.DAACcomment :L2P Core without DT analysis or other ancillary fields; Day, Start Node:Ascending, End Node:Descending; WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value; Refinedlicense :GHRSST and PO.DAAC protocol allow data use as free and open.id :MODIS_A-JPL-L2P-v2019.0naming_authority :org.ghrsstproduct_version :2019.0uuid :f6e1f61d-c4a4-4c17-8354-0c15e12d688bgds_version_id :2.0netcdf_version_id :4.1date_created :20200221T085224Zfile_quality_level :3spatial_resolution :1kmstart_time :20190622T100001Ztime_coverage_start :20190622T100001Zstop_time :20190622T100459Ztime_coverage_end :20190622T100459Znorthernmost_latitude :89.9862southernmost_latitude :66.2723easternmost_longitude :-45.9467westernmost_longitude :152.489source :MODIS sea surface temperature observations for the OBPGplatform :Aquasensor :MODISmetadata_link :http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MODIS_A-JPL-L2P-v2019.0keywords :Oceans > Ocean Temperature > Sea Surface Temperaturekeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordsstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventiongeospatial_lat_units :degrees_northgeospatial_lat_resolution :0.01geospatial_lon_units :degrees_eastgeospatial_lon_resolution :0.01acknowledgment :The MODIS L2P sea surface temperature data are sponsored by NASAcreator_name :Ed Armstrong, JPL PO.DAACcreator_email :edward.m.armstrong@jpl.nasa.govcreator_url :http://podaac.jpl.nasa.govproject :Group for High Resolution Sea Surface Temperaturepublisher_name :The GHRSST Project Officepublisher_url :http://www.ghrsst.orgpublisher_email :ghrsst-po@nceo.ac.ukprocessing_level :L2Pcdm_data_type :swathstartDirection :AscendingendDirection :Descendingday_night_flag :Day\n\n\nLet’s make a quick plot to take a look at the sea_surface_temperature variable.\n\nmodis_ds.sea_surface_temperature.plot() ;\n\n\n\n\n\n\nPlot MODIS and ICESat-2 data on a map\n\nmap_proj = ccrs.NorthPolarStereo()\n\nfig = plt.figure(figsize=(10,5))\nax = fig.add_subplot(projection=map_proj)\nax.coastlines()\n\n# Plot MODIS sst, save object as sst_img, so we can add colorbar\nsst_img = ax.pcolormesh(modis_ds.lon, modis_ds.lat, modis_ds.sea_surface_temperature[0,:,:], \n vmin=240, vmax=270, # Set max and min values for plotting\n cmap='viridis', shading='auto', # shading='auto' to avoid warning\n transform=ccrs.PlateCarree()) # coords are lat,lon but map if NPS \n\n# Plot IS2 surface height \nis2_img = ax.scatter(is2_ds.longitude, is2_ds.latitude,\n c=is2_ds.height_segment_height, \n vmax=1.5, # Set max height to plot\n cmap='Reds', alpha=0.6, s=2,\n transform=ccrs.PlateCarree())\n\n# Add colorbars\nfig.colorbar(sst_img, label='MODIS SST (K)')\nfig.colorbar(is2_img, label='ATL07 Height (m)')\n\n\n<matplotlib.colorbar.Colorbar at 0x7fb3944adb50>\n\n\n\n\n\n\n\nExtract SST coincident with ICESat-2 track\nThe MODIS SST is swath data, not a regularly-spaced grid of sea surface temperatures. ICESat-2 sea surface heights are irregularly spaced segments along one ground-track traced by the ATLAS instrument on-board ICESat-2. Fortunately, pyresample allows us to resample swath data.\npyresample has many resampling methods. We’re going to use the nearest neighbour resampling method, which is implemented using a k-dimensional tree algorithm or K-d tree. K-d trees are data structures that improve search efficiency for large data sets.\nThe first step is to define the geometry of the ICESat-2 and MODIS data. To do this we use the latitudes and longitudes of the datasets.\n\nis2_geometry = pyresample.SwathDefinition(lons=is2_ds.longitude,\n lats=is2_ds.latitude)\n\n\nmodis_geometry = pyresample.SwathDefinition(lons=modis_ds.lon, lats=modis_ds.lat)\n\nWe then implement the resampling method, passing the two geometries we have defined, the data array we want to resample - in this case sea surface temperature, and a search radius. The resampling method expects a numpy.Array rather than an xarray.DataArray, so we use values to get the data as a numpy.Array.\nWe set the search radius to 1000 m. The MODIS data is nominally 1km spacing.\n\nsearch_radius=1000.\nfill_value = np.nan\nis2_sst = pyresample.kd_tree.resample_nearest(\n modis_geometry,\n modis_ds.sea_surface_temperature.values,\n is2_geometry,\n search_radius,\n fill_value=fill_value\n)\n\n\nis2_sst\n\narray([263.375, 263.375, 263.375, ..., nan, nan, nan],\n dtype=float32)\n\n\n\nis2_ds['sea_surface_temperature'] = xr.DataArray(is2_sst, dims='segment')\nis2_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (segment: 235584)\nDimensions without coordinates: segment\nData variables:\n latitude (segment) float64 82.38 82.38 82.38 ... 72.61 72.61\n longitude (segment) float64 -55.11 -55.11 ... 145.1 145.1\n delta_time (segment) float64 4.642e+07 4.642e+07 ... 4.642e+07\n height_segment_height (segment) float32 nan nan nan ... -0.4335 -0.4463\n sea_surface_temperature (segment) float32 263.4 263.4 263.4 ... nan nan nanxarray.DatasetDimensions:segment: 235584Coordinates: (0)Data variables: (5)latitude(segment)float6482.38 82.38 82.38 ... 72.61 72.61contentType :referenceInformationcoordinates :delta_time longitudedescription :Latitude, WGS84, North=+, Lat of segment centerlong_name :Latitudesource :ATBD, section 4.4standard_name :latitudeunits :degrees_northvalid_max :90.0valid_min :-90.0array([82.38431982, 82.38431982, 82.38431982, ..., 72.60984638,\n 72.60977493, 72.60970985])longitude(segment)float64-55.11 -55.11 ... 145.1 145.1contentType :referenceInformationcoordinates :delta_time latitudedescription :Longitude, WGS84, East=+,Lon of segment centerlong_name :Longitudesource :ATBD, section 4.4standard_name :longitudeunits :degrees_eastvalid_max :180.0valid_min :-180.0array([-55.10896068, -55.10896068, -55.10896068, ..., 145.05396164,\n 145.05392851, 145.05389832])delta_time(segment)float644.642e+07 4.642e+07 ... 4.642e+07CLASS :DIMENSION_SCALENAME :gt1l/sea_ice_segments/delta_timeREFERENCE_LIST :[(<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)]contentType :physicalMeasurementcoordinates :latitude longitudedescription :Number of GPS seconds since the ATLAS SDP epoch. The ATLAS Standard Data Products (SDP) epoch offset is defined within /ancillary_data/atlas_sdp_gps_epoch as the number of GPS seconds between the GPS epoch (1980-01-06T00:00:00.000000Z UTC) and the ATLAS SDP epoch. By adding the offset contained within atlas_sdp_gps_epoch to delta time parameters, the time in gps_seconds relative to the GPS epoch can be computed.long_name :Elapsed GPS secondssource :telemetrystandard_name :timeunits :seconds since 2018-01-01array([46419293.64266939, 46419293.64266939, 46419293.64266939, ...,\n 46419681.87646231, 46419681.87759533, 46419681.87862704])height_segment_height(segment)float32nan nan nan ... -0.4335 -0.4463_FillValue :3.4028235e+38contentType :referenceInformationcoordinates :../delta_time ../latitude ../longitudedescription :Mean height from along-track segment fit detremined by the sea ice algorithm. The sea ice height is relative to the tide-free MSS.long_name :height of segment surfacesource :ATBD, section 4.2.2.4units :metersarray([ nan, nan, nan, ..., -0.46550068,\n -0.43347716, -0.4462675 ], dtype=float32)sea_surface_temperature(segment)float32263.4 263.4 263.4 ... nan nan nanarray([263.375, 263.375, 263.375, ..., nan, nan, nan],\n dtype=float32)Attributes: (0)\n\n\n\n\nPlot SST and Height along track\nThis is a quick plot of the extracted data. We’re using matplotlib so we can use latitude as the x-value:\n\nis2_ds = is2_ds.set_coords(['latitude'])\n\nfig, ax1 = plt.subplots(figsize=(15, 7))\nax1.set_xlim(82.,88.)\nax1.plot(is2_ds.latitude, is2_ds.sea_surface_temperature, \n color='orange', label='SST', zorder=3)\nax1.set_ylabel('SST (K)')\n\nax2 = ax1.twinx()\nax2.plot(is2_ds.latitude, is2_ds.height_segment_height, label='Height')\nax2.set_ylabel('Height (m)')\n\nfig.legend()\n\n<matplotlib.legend.Legend at 0x7fb39fcd8040>" }, { - "objectID": "examples/index.html", - "href": "examples/index.html", - "title": "Example Workflows", + "objectID": "tutorials/fair-workflow-geoweaver-demo.html", + "href": "tutorials/fair-workflow-geoweaver-demo.html", + "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", "section": "", - "text": "This section contains end-to-end example workflows for processing and analyzing data in the cloud.\nE.g. Here are some examples for Kerchunk!" + "text": "PyGeoweaver package is a Python wrapper of the GeoWeaver. This package is designed for Jupyter users to be able to directly use Geoweaver in Jupyter notebook or JupyterLab (JupyterHub). ## 1. Installing PyGeoweaver You can either install pygeoweaver using pip or build from source\n!pip install pygeoweaver\n\nRequirement already satisfied: pygeoweaver in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (0.7.4)\nRequirement already satisfied: requests==2.28.2 in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (from pygeoweaver) (2.28.2)\nRequirement already satisfied: setuptools>=61.0 in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (from pygeoweaver) (68.0.0)\nRequirement already satisfied: pydantic==1.10.9 in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (from pygeoweaver) (1.10.9)\nRequirement already satisfied: typing-extensions>=4.2.0 in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (from pydantic==1.10.9->pygeoweaver) (4.6.3)\nRequirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (from requests==2.28.2->pygeoweaver) (1.26.16)\nRequirement already satisfied: certifi>=2017.4.17 in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (from requests==2.28.2->pygeoweaver) (2023.5.7)\nRequirement already satisfied: idna<4,>=2.5 in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (from requests==2.28.2->pygeoweaver) (3.4)\nRequirement already satisfied: charset-normalizer<4,>=2 in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (from requests==2.28.2->pygeoweaver) (3.1.0)\nThe gw.start() function starts the Geoweaver Application on background to serve requests\nimport pygeoweaver as gw\ngw.start()\n\nStop running Geoweaver if any..\nCheck Java..\nStart Geoweaver..\nSuccess: Geoweaver is up" }, { - "objectID": "appendix/index.html", - "href": "appendix/index.html", - "title": "Appendix", - "section": "", - "text": "Extra resources for Earthdata Cloud." + "objectID": "tutorials/fair-workflow-geoweaver-demo.html#reset-passwords-optional-if-you-already-did", + "href": "tutorials/fair-workflow-geoweaver-demo.html#reset-passwords-optional-if-you-already-did", + "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", + "section": "2. Reset Passwords (optional if you already did)", + "text": "2. Reset Passwords (optional if you already did)\nThis password will be used to authenticate the process runs on localhost (a.k.a the default host with id 100001).\n\ngw.reset_password()\n\nEnter password: ········\nRe-enter password: ········\n\n\nNOTE: Password updated." }, { - "objectID": "appendix/index.html#about-the-cloud", - "href": "appendix/index.html#about-the-cloud", - "title": "Appendix", - "section": "About the Cloud", - "text": "About the Cloud\nMaybe you’ve heard that NASA Earthdata is “moving to the cloud” but you want to know why. You can read the details of the Earthdata Cloud Evolution, but here we summarize the benefits of the cloud and additional resources on its use and history. In short, the cloud will make the download of data unnecessary, allow for data processing and manipulation without new software purchases or installation, and, ultimately, reduce the amount of time it takes to get the data needed to do science.\n\nAmazon Web Services\nNASA’s Office of the Chief Information Officer chose Amazon Web Services (AWS) as the source of general-purpose cloud services (but some areas within NASA are working with Google Earth Engine (GEE) to make NASA data accessible in the GEE cloud-based analysis platform). The following resources provide a background on AWS, but much of the information is relevant to folks who want to develop in the cloud rather than simply access data. Remember, all NASA’s science information (including the algorithms, metadata, and documentation associated with science mission data) must be freely available to the public. This means that anyone, anywhere in the world, can access NASA Earth science data without restriction. However, advanced cloud operations could require a user to set-up their own cloud account through AWS or another cloud provider.\n\nCloud Primer for Amazon Web Services This primer provides step-by-step tutorials on how to get started in the AWS cloud.\nWhat is AWS Amazon Web Services is the world’s most comprehensive and broadly adopted cloud, offering over 200 fully featured services from data centers globally.\n\n\n\nCloud Optimized Data Formats\nTraditional file formats can easily be migrated to the cloud, but serving or processing the data from the cloud is inefficient and often requires that the data be downloaded and then translated to another format and stored in memory. Cloud optimized formats are being developed to better serve analysis-in-place workflows that make the cloud so beneficial to science users.\n\nCloud-Optimized Format Study The cloud infrastructure provides a number of capabilities that can dramatically improve access and use of Earth Observation data. However, in many cases, data may need to be reorganized and/or reformatted in order to make them tractable to support cloud-native analysis and access patterns. The purpose of this study is to examine different formats for storing data on the cloud.\nCloud Optimized GeoTIFF A Cloud Optimized GeoTIFF is a regular GeoTIFF file with an internal organization that enables more efficient workflows on the cloud. It does this by leveraging the ability of clients issuing ​HTTP GET range requests to ask for just the parts of a file they need.\nCloud Optimized Formats: NetCDF-as-Zarr Optimizations and Next Steps Building on the work by USGS/HDF to access netCDF as Zarr, the authors found that a sidecar metadata record that includes byte offsets provides users “access HDF5 format data as efficiently as Zarr format data using the Zarr library.” In other words, users can gain the cloud-optimized performance of Zarr while retaining the archival benefits of NetCDF4." + "objectID": "tutorials/fair-workflow-geoweaver-demo.html#import-workflow-earthdata-cloud-clinic", + "href": "tutorials/fair-workflow-geoweaver-demo.html#import-workflow-earthdata-cloud-clinic", + "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", + "section": "3. Import Workflow : Earthdata Cloud Clinic", + "text": "3. Import Workflow : Earthdata Cloud Clinic\nThis tutorial from Cloud cookbook uses two different direct cloud access & subsetting options available in the Earthdata Cloud. * The earthaccess python library for data search and direct cloud access, followed by xarray subsetting * The Harmony-py python library for direct cloud access & data subsetting\nThe workflow for earthdata cloud clinic for geoweaver is hosted at https://github.com/earth-artificial-intelligence/earth_data_cloud_clinic_geoweaver\n\nimport os\nimport urllib.request\nfrom IPython.display import Image\n\nhome_dir = os.path.expanduser('~')\nfile_url = \"https://github.com/earth-artificial-intelligence/earth_data_cloud_clinic_geoweaver/archive/refs/tags/v0.0.6.zip\"\nfile_path = os.path.join(home_dir, \"kr2novmm4i9i27z5adyj.zip\")\n\nurllib.request.urlretrieve(file_url, file_path)\n\ngw.import_workflow(file_path)\n\nUnzipping /Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj.zip to /Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj/\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/.DS_Store\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/README.md\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/earth_access_authenticate.py\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/earth_data_utils.py\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/install_deps.sh\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/install_packages.sh\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/process.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/retrieve_ssh.py\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/retrieve_sst.py\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/s3_file_storage.py\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/earthdata.ipynb\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/geoweaver_screenshot.png\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/gulf.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/195yXqgSfvPIInJ2gI.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/197a10n0xbrc4rqwve.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/1bZvPF5Z1tytBGDqph.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/39SB7eVc2494ECqik2.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/3KpWI4XE8npoDNFXTQ.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/3ZlO4QZpqjhGyKmyXO.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/43kckrxzh5QbESss0R.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/54jcaiqoso3iree437.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/5dixki8g2scz0sjj56.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/5ou0GVSM1NQnlv7FOB.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/5ucpoaa3e0bbnwyu1n.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/5uzqmkhig32ub23cax.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/6LDOLdZroopQ1UvaWF.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/6hxs1emptzyrmsi0f3.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/8CmlmhGPYckTE5VQnh.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/8nqne2wcyke9t1hy5p.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/Ay4VYNLteCA4TvHHvt.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/B2kMBsZQW3Adxgi1uH.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/BacPAagztwKFogMSWa.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/IbB3mgAjkrED9qC8zw.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/JUM7mZ2Clo86FSJQpS.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/KeJgbxTIefDdZTHTLf.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/KmoUyTblH4DeCQYfWN.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/PNFURuqn5MBZk110Fh.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/QBaTZsjXUzZrYyaSGY.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/TX1TxAhpPjO6dvySl7.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/UbPSHj43zEmCk2k5CR.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/UfT7rXlmUDYmfoP804.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/VeCiz4t1MvIQX4i2yN.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/YBoSD2A5E8pi15Utoi.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/YZGTgiuACdQqM2CObt.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/Z9cOHlCTrfGR4beaLN.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/ZYvkLM99T33qWDzU5l.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/a4t3g4kcmx2j5iggfu.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/b70aawjt8z4isn1dwy.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/b8oEcCvvIU6KaKiYoT.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/bv0wvvajfbrscyj6pm.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/c3lYCsxly0BDlEg6da.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/cf2KWtlaFRUAoEIhAS.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/cn80gdwbo9cmtqfmt4.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/csltVkAvLqfza0jV0i.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/drP4XgYr92dISzae7n.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/e02dq17h7tng220fo1.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/eiXb07m8PuodHs3y9W.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/exp8ncff5wdbitkm8o.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/fbSMCmb9pAMK1bogT0.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/gh2rmy5cq1bc003ak4.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/h26w0736xpdftw6zw1.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/hCz5UKbmmsYoiPeelF.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/iMtwJlVtkG2jGPEJCm.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/jid4vegdc9y0pz1oco.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/ju768g303se38lkt74.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/kX9THeiXUlECn7qnLK.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/kialj0m6av03b04sv3.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/kr2novmm4i9i27z5adyj.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/kr6AXjoMUHFXn74s6B.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/l6yVTXJZi87L8UPfDB.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/l873z9b33henwzrydx.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/lXakuW2z080MDGe8i1.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/lbTt2RL9gtcLVP6xKj.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/lwpewb4927s4e4m49r.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/lxABhFBxbAvzawPiKs.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/mczeu09JHbwVkuAAeC.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/mr9YcFRUnuNyThJb4Q.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/mx1w8k46bha4zi0q8r.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/n4tc1hvnq5eamswpkj.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/nfuxH8eVz896NHRm1m.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/nkx236238gagh5wycu.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/nnyv3p5uhplqv43ml3.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/o0enoj5tgqss95igeb.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/oU0aQLscHakCzO9BOi.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/or6xD9foI8celpyyrB.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/otrr4bjr5vvwob8kzf.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/p5qpQWqza1Fwv0GbsF.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/pYWDe5CUpDNQ9g08Uw.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_1b4ko0.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_2a8mok.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_cgvhiy.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_e57twj.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_nlsntt.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_qmc7xi.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_ru79i2.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_v2ow93.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_wttkvt.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_yvo1ds.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/rRrz0ESWwjNUfd8J4l.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/sRPxDerwGuaBIYhhAO.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/tFbeUeqkws6xyNbueE.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/tXEbbqVKQhjVLLM2f3.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/u6EibxNEZg5Or4d4gZ.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/unlenty6p8qrl03z42.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/v97kvdt3cfum1172y2.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/vC36VuDH7D6OOMF68w.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/vx911STRa1BDoIfOd7.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/wDqF0CmSx2MfoE037z.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/woFrPp1c1EdP5vQQMr.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/wq5bex7m1w1t185yyw.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/wxJMPLZFgS9gqClzF6.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/xU3NHiBe4cXiTolHSR.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/xdc6fso7wkybrk71o7.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/xsUiKqCumWr86iaEFQ.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/xvrhhmdem4lsrax6io.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/y2mnxktf5j68wxy68b.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/yf0JCKUqg9sfzi0Tlq.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/ywrvvwo20iif52rga6.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/zkammrkv9md6gesze8.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/zuwTMUYPw7YXrBjWxj.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/workflow.json\nPrecheck: Done\nWorkflow ID: kr2novmm4i9i27z5adyj\nSuccessfully saved to database. Complete." }, { - "objectID": "appendix/index.html#environments", - "href": "appendix/index.html#environments", - "title": "Appendix", - "section": "Environments", - "text": "Environments\n\nManaging Python Environments This book is intended to introduce students to modern computing software, programming tools, and best practices that are broadly applicable to the analysis and visualization of Earth and Environmental data. This section describes basic programming in the open-source Python language.\nReproducible and upgradable Conda environments with conda-lock\nManaging virtual environments with pyenv\nUnderstanding and Improving Conda’s performance\nThe definitive guide to Python virtual environments with conda\nMaking conda fast again" + "objectID": "tutorials/fair-workflow-geoweaver-demo.html#check-workflow-details", + "href": "tutorials/fair-workflow-geoweaver-demo.html#check-workflow-details", + "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", + "section": "4. Check Workflow Details", + "text": "4. Check Workflow Details\nThe workflow has four processes:\n\ninstall_packages - install required packages\nearth_data_utils - define all the utility functions like auth_earthaccess, get_dataset by xarray, etc\nretrieve_ssh - using earthaccess to query EarthDataCloud for SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205\nretrieve_sst - using earthaccess to query EarthDataCloud for MUR-JPL-L4-GLOB-v4.1\n\n\n\n\nScreen Shot 2023-06-21 at 12.41.43 AM.png\n\n\nCheck workflow id and other metadata using detail_workflow.\n\ngw.detail_workflow(\"kr2novmm4i9i27z5adyj\")\n\nWorkflowId: kr2novmm4i9i27z5adyj\n+----------------------+-------------------------+-------------+--------+---------------+\n| Workflow Id | Name | Description | Owner | Confidential? |\n+----------------------+-------------------------+-------------+--------+---------------+\n| kr2novmm4i9i27z5adyj | earth_data_cloud_clinic | | 111111 | FALSE |\n+----------------------+-------------------------+-------------+--------+---------------+" }, { - "objectID": "appendix/index.html#development-on-aws", - "href": "appendix/index.html#development-on-aws", - "title": "Appendix", - "section": "Development on AWS", - "text": "Development on AWS\n\nPython\n\nPython on AWS Tools, docs, and sample code to develop applications on the AWS cloud.\n\n\n\nR\n\nGetting started with R on Amazon Web Services This guide demonstrates how to use AWS in R with the Paws AWS software development kit.\nR for Cloud Computing This book will help you kick-start analytics on the cloud including chapters on both cloud computing, R, common tasks performed in analytics." + "objectID": "tutorials/fair-workflow-geoweaver-demo.html#run-workflow", + "href": "tutorials/fair-workflow-geoweaver-demo.html#run-workflow", + "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", + "section": "5. Run workflow", + "text": "5. Run workflow\nThe follow line will execute the workflow on the provided host. 100001 means localhost. You can use gw.detail_host(\"100001\") to check the details.\nIt will prompt a password input field to ask for password of the target host. If you don’t know your default password of localhost, please run gw.reset_password().\n\ngw.run_workflow(workflow_id='kr2novmm4i9i27z5adyj', host_list='100001')\n\nEnter password for host - 100001: ········\n\n\nRunning workflow kr2novmm4i9i27z5adyj\nThe workflow has been kicked off.\nHistory Id: svx32gpq6x3xmqkorm\nWaiting for it to finish\n> Start to run process: ru79i2\n> Fnished process: ru79i2 - history: zg6v1hy042k - Status: Done\n> Start to run process: e57twj\n> Fnished process: e57twj - history: o2qf88jyh2f - Status: Done\n> Start to run process: nlsntt\n> Fnished process: nlsntt - history: fk604fwceda - Status: Done\nTotal time cost: 104 seconds\nExecution is over. Final status: Done." }, { - "objectID": "appendix/index.html#additional-coding-resources", - "href": "appendix/index.html#additional-coding-resources", - "title": "Appendix", - "section": "Additional Coding Resources", - "text": "Additional Coding Resources\n\nPython\n\nIntro to Geospatial Raster and Vector Data with Python This tutorial provides an introduction to raster data, and describes how to plot, program, and access satellite imagery using Python.\n\n\n\nR\n\nR for Data Science Online Learning Community The R4DS Online Learning Community is a community of R learners at all skill levels working together to improve their skills.\nThe Environmental Data Science Book This book is a living, open and community-driven online resource to showcase and support the publication of data, research and open-source tools for collaborative, reproducible and transparent Environmental Data Science.\nCU EarthLab’s Earth Data Science This site offers free online courses, tutorials, and tools for earth science using R and Python." + "objectID": "tutorials/fair-workflow-geoweaver-demo.html#check-workflow-history", + "href": "tutorials/fair-workflow-geoweaver-demo.html#check-workflow-history", + "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", + "section": "6. Check workflow history", + "text": "6. Check workflow history\nFrom the log, we can see the workflow completed successfully after running all the four member processes. The final process run’s history id is ohvdmvpfy1m. We can use show_history to check.\n\n## Get Results by Checking History\ngw.show_history(\"fk604fwceda\")\n\n+-------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n| History Id | Status | Begin Time | End Time | Input | Output | Notes |\n+-------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n| fk604fwceda | Done | 06/21/2023 03:30:26 | 06/21/2023 03:31:00 | from earth_data_utils import *\n\ndef search_and_get_sst():\n auth_earthaccess()\n results = earthaccess.search_data(\n short_name=sst_short_name,\n cloud_hosted=True,\n temporal=(\"2021-07-01\", \"2021-07-02\"),\n )\n print(\"found results: \", results)\n return results\n\ndef plot_subset(ds_subset):\n ds_subset.plot(figsize=(12,6), x='lon', y='lat')\n home_dir = os.path.expanduser('~')\n \n file_name = f\"geoweaver_plot_sst.png\"\n save_file_path = os.path.join(home_dir, file_name)\n plt.savefig(save_file_path)\n print(f\"new figure is saved to {save_file_path}\")\n\nwith get_dataset(search_and_get_sst()) as ds:\n subset = ds[\"analysed_sst\"].sel(lat=slice(27, 37), \n lon=slice(-70,-50)).std('time')\n plot_subset(subset)\n\n | You're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 08/19/2023\nUsing .netrc file for EDL\nGranules found: 2\nfound results: [Collection: {'Version': '4.1', 'ShortName': 'MUR-JPL-L4-GLOB-v4.1'}\nSpatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'BoundingRectangles': [{'WestBoundingCoordinate': -180, 'SouthBoundingCoordinate': -90, 'EastBoundingCoordinate': 180, 'NorthBoundingCoordinate': 90}]}}}\nTemporal coverage: {'RangeDateTime': {'EndingDateTime': '2021-07-01T21:00:00.000Z', 'BeginningDateTime': '2021-06-30T21:00:00.000Z'}}\nSize(MB): 0\nData: ['https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20210701090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc'], Collection: {'Version': '4.1', 'ShortName': 'MUR-JPL-L4-GLOB-v4.1'}\nSpatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'BoundingRectangles': [{'WestBoundingCoordinate': -180, 'SouthBoundingCoordinate': -90, 'EastBoundingCoordinate': 180, 'NorthBoundingCoordinate': 90}]}}}\nTemporal coverage: {'RangeDateTime': {'EndingDateTime': '2021-07-02T21:00:00.000Z', 'BeginningDateTime': '2021-07-01T21:00:00.000Z'}}\nSize(MB): 0\nData: ['https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20210702090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc']]\n Opening 2 granules, approx size: 0.0 GB\nQUEUEING TASKS | : 0%| | 0/2 [00:00<?, ?it/s]\nQUEUEING TASKS | : 100%|██████████| 2/2 [00:00<00:00, 1421.56it/s]\nPROCESSING TASKS | : 0%| | 0/2 [00:00<?, ?it/s]\nPROCESSING TASKS | : 50%|█████ | 1/2 [00:00<00:00, 1.51it/s]\nPROCESSING TASKS | : 100%|██████████| 2/2 [00:00<00:00, 2.97it/s]\nPROCESSING TASKS | : 100%|██████████| 2/2 [00:00<00:00, 2.59it/s]\nCOLLECTING RESULTS | : 0%| | 0/2 [00:00<?, ?it/s]\nCOLLECTING RESULTS | : 100%|██████████| 2/2 [00:00<00:00, 22192.08it/s]\nnew figure is saved to /Users/joe/geoweaver_plot_sst.png\n | - |\n+-------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n\n\nFrom the history, we can see the result image location is at /Users/joe/geoweaver_plot_sst.png. Let’s display it here." }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Access_COG.html", - "href": "appendix/appendix-alternate-in-cloud-access/Access_COG.html", - "title": "Access Cloud-Optimized GeoTIFF", - "section": "", - "text": "We will access data for the Harmonized Landsat Sentinel-2 (HLS) Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0 (L30) ([10.5067/HLS/HLSL30.002](https://doi.org/10.5067/HLS/HLSL30.002)) data product. These data are archived and distributed as Cloud Optimized GeoTIFF (COG) files, one file for each spectral band.\nWe will access a single COG file, L30 red band (0.64 – 0.67 μm), from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataarray. This approach leverages S3 native protocols for efficient access to the data." + "objectID": "tutorials/fair-workflow-geoweaver-demo.html#display-sea-surface-temperature-data", + "href": "tutorials/fair-workflow-geoweaver-demo.html#display-sea-surface-temperature-data", + "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", + "section": "7. Display sea surface temperature data", + "text": "7. Display sea surface temperature data\n\ndisplay(Image(filename=os.path.join(home_dir, 'geoweaver_plot_sst.png')))\n\n\n\n\nBingo! The image is successfully retrieved by the workflow and the data in xarray is rendered and saved to a local png file. The workflow runs with no errors and the replication succeeded." }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Access_COG.html#summary", - "href": "appendix/appendix-alternate-in-cloud-access/Access_COG.html#summary", - "title": "Access Cloud-Optimized GeoTIFF", - "section": "", - "text": "We will access data for the Harmonized Landsat Sentinel-2 (HLS) Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0 (L30) ([10.5067/HLS/HLSL30.002](https://doi.org/10.5067/HLS/HLSL30.002)) data product. These data are archived and distributed as Cloud Optimized GeoTIFF (COG) files, one file for each spectral band.\nWe will access a single COG file, L30 red band (0.64 – 0.67 μm), from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataarray. This approach leverages S3 native protocols for efficient access to the data." + "objectID": "tutorials/fair-workflow-geoweaver-demo.html#edit-the-process-to-change-date-range", + "href": "tutorials/fair-workflow-geoweaver-demo.html#edit-the-process-to-change-date-range", + "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", + "section": "8. Edit the process to change date range", + "text": "8. Edit the process to change date range\nReplicating is only the start. We always want to change the code and make it work for our purposes. Here we will demonstrate how to change the code in the workflow.\nFirst, let’s list all the processes in the workflow:\n\ngw.list_processes_in_workflow(workflow_id='kr2novmm4i9i27z5adyj')\n\n\n\n\n\n\n\n\ntitle\nid\n\n\n\n\n0\nearth_data_utils\nwttkvt-GJjq8\n\n\n1\ninstall_packages\nru79i2-VCNCv\n\n\n2\nretrieve_sst\nnlsntt-zMUba\n\n\n3\nretrieve_ssh\ne57twj-ECmak\n\n\n\n\n\n\n\nYou can see the ids are a little different with dashes in the middle of them. That is because we need to distinguish the processes in the workflow as there might be same processes called multiple times in one workflow.\nThe original process id is just the first part. For example, the process id of retrieve_sst is nlsntt.\nCopy that id and use get_code_for_process function to get current code.\n\ngw.get_process_code('nlsntt')\n\n'from earth_data_utils import *\\n\\ndef search_and_get_sst():\\n auth_earthaccess()\\n results = earthaccess.search_data(\\n short_name=sst_short_name,\\n cloud_hosted=True,\\n temporal=(\"2021-07-01\", \"2021-07-02\"),\\n )\\n print(\"found results: \", results)\\n return results\\n\\ndef plot_subset(ds_subset):\\n ds_subset.plot(figsize=(12,6), x=\\'lon\\', y=\\'lat\\')\\n home_dir = os.path.expanduser(\\'~\\')\\n \\n file_name = f\"geoweaver_plot_sst.png\"\\n save_file_path = os.path.join(home_dir, file_name)\\n plt.savefig(save_file_path)\\n print(f\"new figure is saved to {save_file_path}\")\\n\\nwith get_dataset(search_and_get_sst()) as ds:\\n subset = ds[\"analysed_sst\"].sel(lat=slice(27, 37), \\n lon=slice(-70,-50)).std(\\'time\\')\\n plot_subset(subset)\\n\\n'\n\n\nThe code is not formatted correctly. So many \\n are confusing.\nAn easy way is to directly save the code into a local file using function sync.\n\n# write the file to disk to make changes\ngw.sync(process_id='nlsntt', local_path=\".\", direction=\"download\")\n\nProceeding with download\n\nWrote file retrieve_sst.py to .\n\n\nThe step above should create a new python file named retrieve_sst.py in the same folder as this notebook.\nOpen it using any editor and you should see the code as follows:\nfrom earth_data_utils import *\n\ndef search_and_get_sst():\n auth_earthaccess()\n results = earthaccess.search_data(\n short_name=sst_short_name,\n cloud_hosted=True,\n temporal=(\"2021-07-01\", \"2021-07-02\"),\n )\n print(\"found results: \", results)\n return results\n\ndef plot_subset(ds_subset):\n ds_subset.plot(figsize=(12,6), x='lon', y='lat')\n home_dir = os.path.expanduser('~')\n \n file_name = f\"geoweaver_plot_{str(uuid.uuid4())}.png\"\n save_file_path = os.path.join(home_dir, file_name)\n plt.savefig(save_file_path)\n print(f\"new figure is saved to {save_file_path}\")\n\nwith get_dataset(search_and_get_sst()) as ds:\n subset = ds[\"analysed_sst\"].sel(lat=slice(27, 37), \n lon=slice(-70,-50)).std('time')\n plot_subset(subset)\nSuppose we want the SST data of another region, for example, change the lat=slice(27, 37) to lat=slice(37, 47). Make the change in retrieve_sst.py, save and close.\nNext step is to write the new code back into Geoweaver process using the same sync function. This time, the option direction should be upload meaning we are syncing changes from local file to Geoweaver database.\n\n# sync changes back to the workflow after editing the file\ngw.sync(process_id='nlsntt', local_path='retrieve_sst.py', direction=\"upload\")\n\nProceeding with upload\n\nProcess update was successful\n\n\nAfter the step finishes, verify the code change using get_process_code again.\n\ngw.get_process_code('nlsntt')\n\n'from earth_data_utils import *\\n\\ndef search_and_get_sst():\\n auth_earthaccess()\\n results = earthaccess.search_data(\\n short_name=sst_short_name,\\n cloud_hosted=True,\\n temporal=(\"2021-07-01\", \"2021-07-02\"),\\n )\\n print(\"found results: \", results)\\n return results\\n\\ndef plot_subset(ds_subset):\\n ds_subset.plot(figsize=(12,6), x=\\'lon\\', y=\\'lat\\')\\n home_dir = os.path.expanduser(\\'~\\')\\n \\n file_name = f\"geoweaver_plot_sst.png\"\\n save_file_path = os.path.join(home_dir, file_name)\\n plt.savefig(save_file_path)\\n print(f\"new figure is saved to {save_file_path}\")\\n\\nwith get_dataset(search_and_get_sst()) as ds:\\n subset = ds[\"analysed_sst\"].sel(lat=slice(37, 47), \\n lon=slice(-70,-50)).std(\\'time\\')\\n plot_subset(subset)\\n\\n'\n\n\nOk, great! You can see the changes are made as the lat= has become 37, 47 now. The editing process is complete.\nNow, we just need to rerun the workflow again with the new code.\n\ngw.run_workflow(workflow_id='kr2novmm4i9i27z5adyj', host_list='100001')\n\nEnter password for host - 100001: ········\n\n\nRunning workflow kr2novmm4i9i27z5adyj\nThe workflow has been kicked off.\nHistory Id: cf5xnsjounspuhlk9z\nWaiting for it to finish\n> Start to run process: ru79i2\n> Fnished process: ru79i2 - history: arr51wopzbw - Status: Done\n> Start to run process: e57twj\n> Fnished process: e57twj - history: ppyfc2he31g - Status: Done\n> Start to run process: nlsntt\n> Fnished process: nlsntt - history: e1ce5zjdm3y - Status: Done\nTotal time cost: 105 seconds\nExecution is over. Final status: Done.\n\n\nYou can see there is a new history id for process nlsntt: m3t1sdcsl3k. Check its content to find the result png file location:\n\ngw.show_history(\"e1ce5zjdm3y\")\n\n+-------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n| History Id | Status | Begin Time | End Time | Input | Output | Notes |\n+-------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n| e1ce5zjdm3y | Done | 06/21/2023 03:37:55 | 06/21/2023 03:38:33 | from earth_data_utils import *\n\ndef search_and_get_sst():\n auth_earthaccess()\n results = earthaccess.search_data(\n short_name=sst_short_name,\n cloud_hosted=True,\n temporal=(\"2021-07-01\", \"2021-07-02\"),\n )\n print(\"found results: \", results)\n return results\n\ndef plot_subset(ds_subset):\n ds_subset.plot(figsize=(12,6), x='lon', y='lat')\n home_dir = os.path.expanduser('~')\n \n file_name = f\"geoweaver_plot_sst.png\"\n save_file_path = os.path.join(home_dir, file_name)\n plt.savefig(save_file_path)\n print(f\"new figure is saved to {save_file_path}\")\n\nwith get_dataset(search_and_get_sst()) as ds:\n subset = ds[\"analysed_sst\"].sel(lat=slice(37, 47), \n lon=slice(-70,-50)).std('time')\n plot_subset(subset)\n\n | You're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 08/19/2023\nUsing .netrc file for EDL\nGranules found: 2\nfound results: [Collection: {'Version': '4.1', 'ShortName': 'MUR-JPL-L4-GLOB-v4.1'}\nSpatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'BoundingRectangles': [{'WestBoundingCoordinate': -180, 'SouthBoundingCoordinate': -90, 'EastBoundingCoordinate': 180, 'NorthBoundingCoordinate': 90}]}}}\nTemporal coverage: {'RangeDateTime': {'EndingDateTime': '2021-07-01T21:00:00.000Z', 'BeginningDateTime': '2021-06-30T21:00:00.000Z'}}\nSize(MB): 0\nData: ['https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20210701090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc'], Collection: {'Version': '4.1', 'ShortName': 'MUR-JPL-L4-GLOB-v4.1'}\nSpatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'BoundingRectangles': [{'WestBoundingCoordinate': -180, 'SouthBoundingCoordinate': -90, 'EastBoundingCoordinate': 180, 'NorthBoundingCoordinate': 90}]}}}\nTemporal coverage: {'RangeDateTime': {'EndingDateTime': '2021-07-02T21:00:00.000Z', 'BeginningDateTime': '2021-07-01T21:00:00.000Z'}}\nSize(MB): 0\nData: ['https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20210702090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc']]\n Opening 2 granules, approx size: 0.0 GB\nQUEUEING TASKS | : 0%| | 0/2 [00:00<?, ?it/s]\nQUEUEING TASKS | : 100%|██████████| 2/2 [00:00<00:00, 1568.26it/s]\nPROCESSING TASKS | : 0%| | 0/2 [00:00<?, ?it/s]\nPROCESSING TASKS | : 50%|█████ | 1/2 [00:00<00:00, 1.21it/s]\nPROCESSING TASKS | : 100%|██████████| 2/2 [00:00<00:00, 2.42it/s]\nCOLLECTING RESULTS | : 0%| | 0/2 [00:00<?, ?it/s]\nCOLLECTING RESULTS | : 100%|██████████| 2/2 [00:00<00:00, 21454.24it/s]\nnew figure is saved to /Users/joe/geoweaver_plot_sst.png\n | - |\n+-------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n\n\nThe new png is generated. Visualize it in jupyter.\n\ndisplay(Image(filename=os.path.join(home_dir, 'geoweaver_plot_sst.png')))\n\n\n\n\nNice! The figure looks great with SST texture different from the one displayed above. That means the change of the spatial extent in the process worked great!\nThe reproducing experiment is done." }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Access_COG.html#code", - "href": "appendix/appendix-alternate-in-cloud-access/Access_COG.html#code", - "title": "Access Cloud-Optimized GeoTIFF", - "section": "Code", - "text": "Code\nHere are our recommended approaches for accessing COG data in NASA Earthdata Cloud with code.\n\nImport Packages\n\nPythonR\n\n\nIn Python we can use the earthaccess library.\nTo install the package we’ll run this code from the command line. Note: you can run shell code directly from a Jupyter Notebook cell by adding a !: !conda install.\n# Install earthaccess\nconda install -c conda-forge earthaccess\nimport earthaccess\nimport requests\nimport os\nimport boto3\nfrom osgeo import gdal\nimport rasterio as rio\nfrom rasterio.session import AWSSession\nimport rioxarray\nimport hvplot.xarray\nimport holoviews as hv\n\n#From Mahsa's tutorial in main:\n#import os\n#from osgeo import gdal\n#import rasterio as rio\n#import rioxarray\n#import hvplot.xarray\n#import holoviews as hv\n\n\nlibrary(rgdal)\nlibrary(raster)\nlibrary(terra)" + "objectID": "tutorials/fair-workflow-geoweaver-demo.html#check-history-of-specific-processes-or-workflows", + "href": "tutorials/fair-workflow-geoweaver-demo.html#check-history-of-specific-processes-or-workflows", + "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", + "section": "9. Check history of specific processes or workflows", + "text": "9. Check history of specific processes or workflows\n\n# get run history of the process\ngw.get_process_history('nlsntt')\n\n\n\n\n\n\n\n\nhistory_id\nhistory_input\nhistory_output\nhistory_begin_time\nhistory_end_time\nhistory_notes\nhistory_process\nhost_id\nindicator\n\n\n\n\n0\ne1ce5zjdm3y\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 07:37:55.840\n2023-06-21 07:38:33.137\nNone\nnlsntt\n100001\nDone\n\n\n1\nafz7j3dzavr\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 07:33:45.739\n2023-06-21 07:34:17.617\nNone\nnlsntt\n100001\nDone\n\n\n2\nfk604fwceda\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 07:30:26.691\n2023-06-21 07:31:00.588\nNone\nnlsntt\n100001\nDone\n\n\n3\nuf7ngwa1cnl\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 07:27:23.479\n2023-06-21 07:27:56.093\nNone\nnlsntt\n100001\nDone\n\n\n4\njxyo4sjnw7i4dsvt30\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 07:07:44.443\n2023-06-21 07:08:22.248\nNone\nnlsntt\nNone\nDone\n\n\n5\nm3t1sdcsl3k\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 07:02:58.529\n2023-06-21 07:03:31.363\nNone\nnlsntt\n100001\nDone\n\n\n6\nohvdmvpfy1m\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 04:35:07.128\n2023-06-21 04:35:41.058\nNone\nnlsntt\n100001\nDone\n\n\n7\n55ierovrm5x\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 04:21:51.164\n2023-06-21 04:22:26.277\nNone\nnlsntt\n100001\nDone\n\n\n8\nEcOKe1e6zRlz\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 04:19:36.446\n2023-06-21 04:20:10.986\nNone\nnlsntt\nNone\nDone\n\n\n9\nClldd9m98PAt\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 04:15:33.330\n2023-06-21 04:16:09.377\nNone\nnlsntt\nNone\nDone\n\n\n10\nsAZF2bohiT4g\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 04:10:58.009\n2023-06-21 04:11:29.683\nNone\nnlsntt\nNone\nDone\n\n\n11\nmuhcxxqhzna\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 04:08:44.419\n2023-06-21 04:09:36.507\nNone\nnlsntt\n100001\nDone\n\n\n12\n5c4g1wdrzme\nfrom earth_data_utils import *\\n\\ndef search_a...\nRunning\n2023-06-21 04:07:23.881\n2023-06-21 04:07:35.184\nNone\nnlsntt\n100001\nDone\n\n\n13\nV4V7YE0Q20Qd\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 04:05:26.906\n2023-06-21 04:06:20.076\nNone\nnlsntt\nNone\nDone\n\n\n14\ngH0Y1ntFvvJQ\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 04:01:01.589\n2023-06-21 04:01:57.329\nNone\nnlsntt\nNone\nDone\n\n\n15\nbc8wLVHggQC2\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 03:55:43.750\n2023-06-21 03:56:34.311\nNone\nnlsntt\nNone\nFailed\n\n\n16\nfFuSBuGyhfUh\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 03:44:26.103\n2023-06-21 03:55:39.992\nNone\nnlsntt\nNone\nDone\n\n\n17\nMt9qs1sz3ypu\nfrom earth_data_utils import *\\n\\ndef search_a...\nFile \"/Users/joe/gw-workspace/Mt9qs1sz3ypu/r...\n2023-06-21 03:44:13.505\n2023-06-21 03:44:13.553\nNone\nnlsntt\nNone\nFailed\n\n\n18\ng5wgNPIMKvt6\nfrom earth_data_utils import *\\n\\ndef search_a...\nFile \"/Users/joe/gw-workspace/g5wgNPIMKvt6/r...\n2023-06-21 03:43:54.668\n2023-06-21 03:43:54.724\nNone\nnlsntt\nNone\nFailed\n\n\n19\nR1pmcdWqLp6V\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 03:40:48.248\n2023-06-21 03:41:17.846\nNone\nnlsntt\nNone\nFailed\n\n\n20\nD5poheZLB3st\nfrom earth_data_utils import *\\n\\ndef search_a...\nTraceback (most recent call last):\\n File \"/U...\n2023-06-21 03:40:35.390\n2023-06-21 03:40:41.692\nNone\nnlsntt\nNone\nFailed\n\n\n21\n5lqwvos3wb6\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 03:33:59.859\n2023-06-21 03:34:27.842\nNone\nnlsntt\n100001\nFailed\n\n\n22\nXYWhBOUtZcPG\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 03:25:02.150\n2023-06-21 03:25:36.884\nNone\nnlsntt\nNone\nFailed\n\n\n23\n4NoU0dDwnwmb\nfrom earth_data_utils import *\\n\\ndef search_a...\nTraceback (most recent call last):\\n File \"/U...\n2023-06-21 03:24:10.798\n2023-06-21 03:24:17.863\nNone\nnlsntt\nNone\nFailed\n\n\n24\n4AheIx9gmzVw\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 03:15:01.301\n2023-06-21 03:15:33.319\nNone\nnlsntt\nNone\nFailed\n\n\n25\nGNDa18SBhJTV\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 03:11:51.951\n2023-06-21 03:12:27.523\nNone\nnlsntt\nNone\nFailed\n\n\n\n\n\n\n\n\ngw.get_workflow_history('kr2novmm4i9i27z5adyj')\n\n\n\n\n\n\n\n\nhistory_id\nhistory_input\nhistory_output\nhistory_begin_time\nhistory_end_time\nhistory_notes\nhistory_process\nhost_id\nindicator\n\n\n\n\n0\ncf5xnsjounspuhlk9z\nwttkvt-GJjq8;ru79i2-VCNCv;nlsntt-zMUba;e57twj-...\nfwu7qad3kq6;arr51wopzbw;e1ce5zjdm3y;ppyfc2he31g;\n2023-06-21 07:36:46.153\n2023-06-21 07:37:55.840000000\nNone\nkr2novmm4i9i27z5adyj\n100001;\nDone\n\n\n1\ndad8g3gkb76stetwix\nwttkvt-GJjq8;ru79i2-VCNCv;nlsntt-zMUba;e57twj-...\n1c5sjmcrgnj;fx19g9buif9;afz7j3dzavr;jlmxs448gc7;\n2023-06-21 07:32:27.062\n2023-06-21 07:33:45.739000064\nNone\nkr2novmm4i9i27z5adyj\n100001;\nDone\n\n\n2\nsvx32gpq6x3xmqkorm\nwttkvt-GJjq8;ru79i2-VCNCv;nlsntt-zMUba;e57twj-...\nzle2vcmqtkh;zg6v1hy042k;fk604fwceda;o2qf88jyh2f;\n2023-06-21 07:29:18.002\n2023-06-21 07:30:26.691000064\nNone\nkr2novmm4i9i27z5adyj\n100001;\nDone\n\n\n3\nb70aawjt8z4isn1dwy\nwttkvt-GJjq8;ru79i2-VCNCv;nlsntt-zMUba;e57twj-...\n617pj1d8nis;zzcxtngjbdu;uf7ngwa1cnl;agb7rrec49u;\n2023-06-21 07:26:09.740\n2023-06-21 07:26:14.760999936\nNone\nkr2novmm4i9i27z5adyj\n100001;\nRunning\n\n\n4\nbv0wvvajfbrscyj6pm\nwttkvt-GJjq8;ru79i2-VCNCv;nlsntt-zMUba;e57twj-...\n1h6mwhjgffr;ced80h6vr3r;m3t1sdcsl3k;ed1lczm7cgh;\n2023-06-21 07:01:45.800\n2023-06-21 07:02:58.528999936\nNone\nkr2novmm4i9i27z5adyj\n100001;\nDone\n\n\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n\n\n94\nnfuxH8eVz896NHRm1m\nwttkvt-GJjq8;yvo1ds-96vns;cgvhiy-BgnXM;v2ow93-...\nrm0v06cmvuc;8nq18ogdwdi;jhzwkhohmz7;6o2p0699ri...\n2023-06-08 16:08:37.776\n2023-06-08 16:08:43.665999872\nNone\nkr2novmm4i9i27z5adyj\njf7wuu;\nFailed\n\n\n95\nVeCiz4t1MvIQX4i2yN\nwttkvt-GJjq8;yvo1ds-96vns;cgvhiy-BgnXM;v2ow93-...\n15jtnmmliz8;sj2llldbybq;alm4h4h5irw;fs47znp6yg...\n2023-06-08 16:08:06.441\n2023-06-08 16:08:14.723000064\nNone\nkr2novmm4i9i27z5adyj\njf7wuu;\nFailed\n\n\n96\ntFbeUeqkws6xyNbueE\nwttkvt-GJjq8;yvo1ds-96vns;cgvhiy-BgnXM;v2ow93-...\n5gi96g5psgp;7ikdfl3nkyj;zftnvonktj6;aipjqmvzj5...\n2023-06-08 16:06:07.658\nNaT\nNone\nkr2novmm4i9i27z5adyj\njf7wuu;\nStopped\n\n\n97\nPNFURuqn5MBZk110Fh\nwttkvt-GJjq8;yvo1ds-96vns;cgvhiy-BgnXM;v2ow93-...\nro72gckii7a;dnqed48moze;xb7v9iap72l;8heewks755...\n2023-06-08 16:05:50.800\nNaT\nNone\nkr2novmm4i9i27z5adyj\njf7wuu;\nStopped\n\n\n98\nwxJMPLZFgS9gqClzF6\nwttkvt-GJjq8;yvo1ds-96vns;cgvhiy-BgnXM;v2ow93-...\n9a06gtkzws7;4z8l2w2pq5n;2z9k699i4su;w8f5ftv7f9...\n2023-06-08 16:05:15.441\nNaT\nNone\nkr2novmm4i9i27z5adyj\njf7wuu;\nStopped\n\n\n\n\n99 rows × 9 columns" }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Access_COG.html#workspace-environment-setup", - "href": "appendix/appendix-alternate-in-cloud-access/Access_COG.html#workspace-environment-setup", - "title": "Access Cloud-Optimized GeoTIFF", - "section": "Workspace Environment Setup", - "text": "Workspace Environment Setup\n\nPythonR\n\n\nFor this exercise, we are going to open up a context manager for the notebook using the rasterio.env module to store the required GDAL configurations we need to access the data from Earthdata Cloud. While the context manager is open (rio_env.__enter__()) we will be able to run the open or get data commands that would typically be executed within a with statement, thus allowing us to more freely interact with the data. We’ll close the context (rio_env.__exit__()) at the end of the notebook.\nGDAL environment variables must be configured to access COGs from Earthdata Cloud. Geospatial data access Python packages like rasterio and rioxarray depend on GDAL, leveraging GDAL’s “Virtual File Systems” to read remote files. GDAL has a lot of environment variables that control it’s behavior. Changing these settings can mean the difference being able to access a file or not. They can also have an impact on the performance.\nrio_env = rio.Env(GDAL_DISABLE_READDIR_ON_OPEN='TRUE',\n GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/cookies.txt'),\n GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/cookies.txt'))\nrio_env.__enter__()\n\n\nSet up rgdal configurations to access the cloud assets that we are interested in. You can learn more about these configuration options here.\nsetGDALconfig(c(\"GDAL_HTTP_UNSAFESSL=YES\",\n \"GDAL_HTTP_COOKIEFILE=.rcookies\",\n \"GDAL_HTTP_COOKIEJAR=.rcookies\",\n \"GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR\",\n \"CPL_VSIL_CURL_ALLOWED_EXTENSIONS=TIF\"))\n\n\n\nIn this example we’re interested in the HLS L30 data collection from NASA’s LP DAAC in Earthdata Cloud. Below we specify the HTTPS URL to the data asset in Earthdata Cloud. This URL can be found via Earthdata Search or programmatically through the CMR and CMR-STAC APIs.\n\nPythonR\n\n\nhttps_url = 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T11SQA.2021333T181532.v2.0/HLS.L30.T11SQA.2021333T181532.v2.0.B04.tif'\n\n\nPlease note that in R, we need to add /vsicurl/ manually to the COG file URL.\nhttps_url <- '/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T11SQA.2021333T181532.v2.0/HLS.L30.T11SQA.2021333T181532.v2.0.B04.tif'" + "objectID": "tutorials/fair-workflow-geoweaver-demo.html#directly-run-local-python-process-using-pygeoweaver", + "href": "tutorials/fair-workflow-geoweaver-demo.html#directly-run-local-python-process-using-pygeoweaver", + "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", + "section": "10. Directly run local python process using PyGeoweaver", + "text": "10. Directly run local python process using PyGeoweaver\n\ngw.run_process(host_id='100001', process_id='nlsntt', sync_path='retrieve_sst.py')\n\nEnter password for host - 100001: ········\n\n\nUpdating code on workflow with the given file path.\n\nStaring process nlsntt\nExecution finished\nTotal time cost: 41 seconds\n+--------------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n| History Id | Status | Begin Time | End Time | Input | Output | Notes |\n+--------------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n| orznud2v2ugsyqfjlj | Done | 06/21/2023 03:40:26 | 06/21/2023 03:41:00 | from earth_data_utils import *\n\ndef search_and_get_sst():\n auth_earthaccess()\n results = earthaccess.search_data(\n short_name=sst_short_name,\n cloud_hosted=True,\n temporal=(\"2021-07-01\", \"2021-07-02\"),\n )\n print(\"found results: \", results)\n return results\n\ndef plot_subset(ds_subset):\n ds_subset.plot(figsize=(12,6), x='lon', y='lat')\n home_dir = os.path.expanduser('~')\n \n file_name = f\"geoweaver_plot_sst.png\"\n save_file_path = os.path.join(home_dir, file_name)\n plt.savefig(save_file_path)\n print(f\"new figure is saved to {save_file_path}\")\n\nwith get_dataset(search_and_get_sst()) as ds:\n subset = ds[\"analysed_sst\"].sel(lat=slice(37, 47), \n lon=slice(-70,-50)).std('time')\n plot_subset(subset)\n\n | You're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 08/19/2023\nUsing .netrc file for EDL\nGranules found: 2\nfound results: [Collection: {'Version': '4.1', 'ShortName': 'MUR-JPL-L4-GLOB-v4.1'}\nSpatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'BoundingRectangles': [{'WestBoundingCoordinate': -180, 'SouthBoundingCoordinate': -90, 'EastBoundingCoordinate': 180, 'NorthBoundingCoordinate': 90}]}}}\nTemporal coverage: {'RangeDateTime': {'EndingDateTime': '2021-07-01T21:00:00.000Z', 'BeginningDateTime': '2021-06-30T21:00:00.000Z'}}\nSize(MB): 0\nData: ['https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20210701090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc'], Collection: {'Version': '4.1', 'ShortName': 'MUR-JPL-L4-GLOB-v4.1'}\nSpatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'BoundingRectangles': [{'WestBoundingCoordinate': -180, 'SouthBoundingCoordinate': -90, 'EastBoundingCoordinate': 180, 'NorthBoundingCoordinate': 90}]}}}\nTemporal coverage: {'RangeDateTime': {'EndingDateTime': '2021-07-02T21:00:00.000Z', 'BeginningDateTime': '2021-07-01T21:00:00.000Z'}}\nSize(MB): 0\nData: ['https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20210702090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc']]\n Opening 2 granules, approx size: 0.0 GB\nQUEUEING TASKS | : 0%| | 0/2 [00:00<?, ?it/s]\nQUEUEING TASKS | : 100%|██████████| 2/2 [00:00<00:00, 2267.80it/s]\nPROCESSING TASKS | : 0%| | 0/2 [00:00<?, ?it/s]\nPROCESSING TASKS | : 50%|█████ | 1/2 [00:00<00:00, 1.53it/s]\nPROCESSING TASKS | : 100%|██████████| 2/2 [00:00<00:00, 2.71it/s]\nCOLLECTING RESULTS | : 0%| | 0/2 [00:00<?, ?it/s]\nCOLLECTING RESULTS | : 100%|██████████| 2/2 [00:00<00:00, 16131.94it/s]\nnew figure is saved to /Users/joe/geoweaver_plot_sst.png\n | - |\n+--------------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n\n\nUse the same code to visualize the result in a plot.\n\ndisplay(Image(filename=os.path.join(home_dir, 'geoweaver_plot_sst.png')))\n\n\n\n\nThis image has the same texture as we got in the previous workflow run, and it means running local python directly in pygeoweaver will have the same outputs and won’t impact the consistency.\nPyGeoWeaver offers significant benefits over manual management of geospatial processes. It ensures that result images remain consistent regardless of file location changes, enables seamless integration with your local Python file, and provides a repeatable and automated process. By utilizing PyGeoWeaver, you can achieve consistent, reproducible, and efficient geospatial analysis, enhancing your productivity and reducing the burden of manual workflow management." }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Access_COG.html#https-data-access", - "href": "appendix/appendix-alternate-in-cloud-access/Access_COG.html#https-data-access", - "title": "Access Cloud-Optimized GeoTIFF", - "section": "HTTPS Data Access", - "text": "HTTPS Data Access\nRead in the HLS HTTPS URL for the L30 red band (0.64 – 0.67 μm) into our workspace. Note that, accessing files in the cloud requires you to authenticate using your NASA Earthdata Login account meaning a proper netrc file needs to be set up.\n\nPythonRMatlabCommand Line\n\n\nWe will authenticate below using a netrc file. See the (TBD) appendix for more information on netrc setup.\nauth = Auth().login(strategy=\"netrc\")\n# are we authenticated?\nif not auth.authenticated:\n # ask for credentials and persist them in a .netrc file\n auth.login(strategy=\"interactive\", persist=True)\n\n# The Store class will let us download data from NASA directly\nstore = Store(auth)\n\nWorking with the URLs directly\nIf we choose, we can use earthaccess to grab the file’s URLs and then access them with another library. Getting the links to our data is quiet simple with the data_links() method on each of the results. See the previous Find Data How-To for more information on how to discover datasets of interest.\n#Searching over a small plot in Nebraska, USA over two weeks in September 2022\ngranules = DataGranules().concept_id(\"C2021957657-LPCLOUD\").temporal(\"2022-09-10\",\"2022-09-24\").bounding_box(-101.67271,41.04754,-101.65344,41.06213).get()\nprint(len(granules))\ngranules[0].data_links(access=\"direct\")\n\n\nGet Temporary AWS Credentials\nDirect S3 access is achieved by passing NASA supplied temporary credentials to AWS so we can interact with S3 objects from applicable Earthdata Cloud buckets. For now, each NASA DAAC has different AWS credentials endpoints. Below are some of the credential endpoints to various DAACs.\nCOMING SOON: We can use the earthaccess store class to pass these credentials directly to Boto3 without the need to set up this function.\ns3_cred_endpoint = {\n 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n 'gesdisc': 'https://data.gesdisc.earthdata.nasa.gov/s3credentials',\n 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n 'ornldaac': 'https://data.ornldaac.earthdata.nasa.gov/s3credentials',\n 'ghrcdaac': 'https://data.ghrc.earthdata.nasa.gov/s3credentials'\n}\n\ndef get_temp_creds(provider):\n return requests.get(s3_cred_endpoint[provider]).json()\n\ntemp_creds_req = get_temp_creds('lpdaac')\nCreate a boto3 Session object using your temporary credentials. This Session is used to pass credentials and configuration to AWS so we can interact wit S3 objects from applicable buckets.\nsession = boto3.Session(aws_access_key_id=temp_creds_req['accessKeyId'], \n aws_secret_access_key=temp_creds_req['secretAccessKey'],\n aws_session_token=temp_creds_req['sessionToken'],\n region_name='us-west-2')\nGDAL environment variables must be configured to access COGs in Earthdata Cloud:\nrio_env = rio.Env(AWSSession(session),\n GDAL_DISABLE_READDIR_ON_OPEN='TRUE',\n GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/cookies.txt'),\n GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/cookies.txt'))\nrio_env.__enter__()\n\n\nDirect In-region Access\nRead in the HLS s3 URL for the L30 red band (0.64 – 0.67 μm) into our workspace using rioxarray, an extension of xarray used to read geospatial data. The file is read into Python as an xarray dataarray with a band, x, and y dimension. In this example the band dimension is meaningless, so we’ll use the squeeze() function to remove band as a dimension.\ns3_url = granules[0].data_links(access=\"direct\")[8]\nda = rioxarray.open_rasterio(s3_url)\nda_red = da.squeeze('band', drop=True)\nda_red\nPlot the dataarray, representing the L30 red band, using hvplot.\nda_red.hvplot.image(x='x', y='y', cmap='gray', aspect='equal')\nExit the context manager.\nrio_env.__exit__()\n\n\n\nda_red <- rast(https_url)\nda_red\nThe Convert a SpatRaster object to a Raster object using raster() to be able to use leaflet to plot our data.\nred_raster <- da_red %>% raster()\nred_raster\nThen plot the red band using plot function.\nplot(red_raster)\n\n\nMatlab code coming soon!\n#| echo: true\n# Coming soon!\n\n\nWith wget and curl:\n# Coming soon!" + "objectID": "tutorials/fair-workflow-geoweaver-demo.html#geoweaver-gui-for-creating-and-editing-workflows", + "href": "tutorials/fair-workflow-geoweaver-demo.html#geoweaver-gui-for-creating-and-editing-workflows", + "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", + "section": "11. Geoweaver GUI for creating and editing workflows", + "text": "11. Geoweaver GUI for creating and editing workflows\nThere is something not covered in this tutorial about how to create a new workflow in Geoweaver. We usually do that in the Geoweaver GUI.\nCheck if you can see Geoweaver GUI using the show method like the picture below.\n\n\n\nimage.png\n\n\nIf you’re running Jupyter on your localhost, accessing Geoweaver is like a breeze. If you’re on your localhost, simply type gw.show in a Jupyter cell block or navigate to http://localhost:8070/Geoweaver in your browser, and you’re good to go. However, if you’re running Jupyter on a remote host, a little extra setup is needed to access Geoweaver.\nTo access Geoweaver on a remote host, you’ll need to proxy the URL to a public address. For instance, if you’re using an AWS EC2 instance, replace “localhost” in the URL with the public IP address of your instance. Then enter the modified URL in your browser. Don’t forget to add port 8070 to the inbound rule in your EC2 instance’s security group settings to ensure a smooth connection.\nIf you’re eager to explore Geoweaver’s GUI and dive into detailed tutorials, you can find them at https://esipfed.github.io/Geoweaver/. However, in this notebook, our focus is on demonstrating the incredible capabilities of Geoweaver for replicating and reproducing workflows.\n\ngw.show()\n\n2023-06-21 03:17:27,586 - INFO - enter ipython block\n2023-06-21 03:17:27,587 - WARNING - This only works when the Jupyter is visited from localhost!" }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html", - "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html", - "title": "Accessing Cloud Optimized GeoTIFF (COG) - HTTPS Example", + "objectID": "tutorials/Sea_Level_Rise.html", + "href": "tutorials/Sea_Level_Rise.html", + "title": "Analyzing Sea Level Rise with NASA Earthdata", "section": "", - "text": "In this notebook, we will access data for the Harmonized Landsat Sentinel-2 (HLS) Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0 (L30) ([10.5067/HLS/HLSL30.002](https://doi.org/10.5067/HLS/HLSL30.002)) data product. These data are archived and distributed as Cloud Optimized GeoTIFF (COG) files, one file for each spectral band.\nWe will access a single COG file, L30 red band (0.64 – 0.67 μm), from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataarray. This approach leverages S3 native protocols for efficient access to the data." + "text": "image.png" }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#summary", - "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#summary", - "title": "Accessing Cloud Optimized GeoTIFF (COG) - HTTPS Example", - "section": "", - "text": "In this notebook, we will access data for the Harmonized Landsat Sentinel-2 (HLS) Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0 (L30) ([10.5067/HLS/HLSL30.002](https://doi.org/10.5067/HLS/HLSL30.002)) data product. These data are archived and distributed as Cloud Optimized GeoTIFF (COG) files, one file for each spectral band.\nWe will access a single COG file, L30 red band (0.64 – 0.67 μm), from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataarray. This approach leverages S3 native protocols for efficient access to the data." + "objectID": "tutorials/Sea_Level_Rise.html#overview", + "href": "tutorials/Sea_Level_Rise.html#overview", + "title": "Analyzing Sea Level Rise with NASA Earthdata", + "section": "Overview", + "text": "Overview\nThis tutorial analyzes global sea level rise from satellite altimetry data and compares it to a historic record. We will be reproducing the plot from this infographic: NASA-led Study Reveals the Causes of Sea Level Rise Since 1900." }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#requirements", - "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#requirements", - "title": "Accessing Cloud Optimized GeoTIFF (COG) - HTTPS Example", - "section": "Requirements", - "text": "Requirements\n\n1. AWS instance running in us-west-2\nNASA Earthdata Cloud data in S3 can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region.\n\n\n2. Earthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\n3. netrc File\nYou will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. For additional information see: Authentication for NASA Earthdata." + "objectID": "tutorials/Sea_Level_Rise.html#learning-objectives", + "href": "tutorials/Sea_Level_Rise.html#learning-objectives", + "title": "Analyzing Sea Level Rise with NASA Earthdata", + "section": "Learning Objectives", + "text": "Learning Objectives\n\nSearch for data programmatically using keywords\nAccess data within the AWS cloud using the earthaccess python library\nVisualize sea level rise trends from altimetry datasets and compare against historic records" }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#learning-objectives", - "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#learning-objectives", - "title": "Accessing Cloud Optimized GeoTIFF (COG) - HTTPS Example", - "section": "Learning Objectives", - "text": "Learning Objectives\n\nhow to configure your Python and R work environment to access Cloud Optimized geoTIFF (COG) files\nhow to access HLS COG files\nhow to plot the data" + "objectID": "tutorials/Sea_Level_Rise.html#requirements", + "href": "tutorials/Sea_Level_Rise.html#requirements", + "title": "Analyzing Sea Level Rise with NASA Earthdata", + "section": "Requirements", + "text": "Requirements\n\n1. Compute environment - This notebook can only be run in an AWS Cloud instance running in the us-west-2 region.\n\n\n2. Earthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\nInstall Packages\n\nimport earthaccess\nimport xarray as xr\nfrom pyproj import Geod\nimport numpy as np\nimport hvplot.xarray\nfrom matplotlib import pyplot as plt\nfrom pprint import pprint\n\n\n\n\n\n\n\n\n\n\n\nWe recommend authenticating your Earthdata Login (EDL) information using the earthaccess python package as follows:\n\nauth = earthaccess.login(strategy=\"netrc\") # works if the EDL login already been persisted to a netrc\nif not auth.authenticated:\n # ask for EDL credentials and persist them in a .netrc file\n auth = earthaccess.login(strategy=\"interactive\", persist=True)\n\nYou're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 09/10/2023\nUsing .netrc file for EDL" }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#cloud-optimized-geotiff-cog", - "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#cloud-optimized-geotiff-cog", - "title": "Accessing Cloud Optimized GeoTIFF (COG) - HTTPS Example", - "section": "Cloud Optimized GeoTIFF (COG)", - "text": "Cloud Optimized GeoTIFF (COG)\nUsing Harmonized Landsat Sentinel-2 (HLS) version 2.0\n\nImport Packages\n\nPythonR\n\n\nimport os\nfrom osgeo import gdal\nimport rasterio as rio\nimport rioxarray\nimport hvplot.xarray\nimport holoviews as hv\n\n\nlibrary(rgdal)\nlibrary(raster)\nlibrary(terra)" + "objectID": "tutorials/Sea_Level_Rise.html#search-for-sea-surface-height-data", + "href": "tutorials/Sea_Level_Rise.html#search-for-sea-surface-height-data", + "title": "Analyzing Sea Level Rise with NASA Earthdata", + "section": "Search for Sea Surface Height Data", + "text": "Search for Sea Surface Height Data\nLet’s find the first four collections that match a keyword search for Sea Surface Height and print out the first two.\n\ncollections = earthaccess.collection_query().keyword(\"SEA SURFACE HEIGHT\").cloud_hosted(True).get(4)\n\n# Let's print 2 collections\nfor collection in collections[0:2]:\n # pprint(collection.summary())\n print(pprint(collection.summary()), collection.abstract(), \"\\n\", collection[\"umm\"][\"DOI\"], \"\\n\\n\")\n\n{'cloud-info': {'Region': 'us-west-2',\n 'S3BucketAndObjectPrefixNames': ['podaac-ops-cumulus-public/SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205/',\n 'podaac-ops-cumulus-protected/SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205/'],\n 'S3CredentialsAPIDocumentationURL': 'https://archive.podaac.earthdata.nasa.gov/s3credentialsREADME',\n 'S3CredentialsAPIEndpoint': 'https://archive.podaac.earthdata.nasa.gov/s3credentials'},\n 'concept-id': 'C2270392799-POCLOUD',\n 'file-type': \"[{'Format': 'netCDF-4', 'FormatType': 'Native', \"\n \"'AverageFileSize': 9.7, 'AverageFileSizeUnit': 'MB'}]\",\n 'get-data': ['https://cmr.earthdata.nasa.gov/virtual-directory/collections/C2270392799-POCLOUD',\n 'https://search.earthdata.nasa.gov/search/granules?p=C2270392799-POCLOUD'],\n 'short-name': 'SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205',\n 'version': '2205'}\nNone This dataset provides gridded Sea Surface Height Anomalies (SSHA) above a mean sea surface, on a 1/6th degree grid every 5 days. It contains the fully corrected heights, with a delay of up to 3 months. The gridded data are derived from the along-track SSHA data of TOPEX/Poseidon, Jason-1, Jason-2, Jason-3 and Jason-CS (Sentinel-6) as reference data from the level 2 along-track data found at https://podaac.jpl.nasa.gov/dataset/MERGED_TP_J1_OSTM_OST_CYCLES_V51, plus ERS-1, ERS-2, Envisat, SARAL-AltiKa, CryoSat-2, Sentinel-3A, Sentinel-3B, depending on the date, from the RADS database. The date given in the grid files is the center of the 5-day window. The grids were produced from altimeter data using Kriging interpolation, which gives best linear prediction based upon prior knowledge of covariance. \n {'DOI': '10.5067/SLREF-CDRV3', 'Authority': 'https://doi.org'} \n\n\n{'cloud-info': {'Region': 'us-west-2',\n 'S3BucketAndObjectPrefixNames': ['nsidc-cumulus-prod-protected/ATLAS/ATL21/003',\n 'nsidc-cumulus-prod-public/ATLAS/ATL21/003'],\n 'S3CredentialsAPIDocumentationURL': 'https://data.nsidc.earthdatacloud.nasa.gov/s3credentialsREADME',\n 'S3CredentialsAPIEndpoint': 'https://data.nsidc.earthdatacloud.nasa.gov/s3credentials'},\n 'concept-id': 'C2753316241-NSIDC_CPRD',\n 'file-type': \"[{'FormatType': 'Native', 'Format': 'HDF5', \"\n \"'FormatDescription': 'HTTPS'}]\",\n 'get-data': ['https://search.earthdata.nasa.gov/search?q=ATL21+V003'],\n 'short-name': 'ATL21',\n 'version': '003'}\nNone This data set contains daily and monthly gridded polar sea surface height (SSH) anomalies, derived from the along-track ATLAS/ICESat-2 L3A Sea Ice Height product (ATL10, V6). The ATL10 product identifies leads in sea ice and establishes a reference sea surface used to estimate SSH in 10 km along-track segments. ATL21 aggregates the ATL10 along-track SSH estimates and computes daily and monthly gridded SSH anomaly in NSIDC Polar Stereographic Northern and Southern Hemisphere 25 km grids. \n {'DOI': '10.5067/ATLAS/ATL21.003'}" }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#workspace-environment-setup", - "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#workspace-environment-setup", - "title": "Accessing Cloud Optimized GeoTIFF (COG) - HTTPS Example", - "section": "Workspace Environment Setup", - "text": "Workspace Environment Setup\n\nPythonR\n\n\nFor this exercise, we are going to open up a context manager for the notebook using the rasterio.env module to store the required GDAL configurations we need to access the data from Earthdata Cloud. While the context manager is open (rio_env.__enter__()) we will be able to run the open or get data commands that would typically be executed within a with statement, thus allowing us to more freely interact with the data. We’ll close the context (rio_env.__exit__()) at the end of the notebook.\nGDAL environment variables must be configured to access COGs from Earthdata Cloud. Geospatial data access Python packages like rasterio and rioxarray depend on GDAL, leveraging GDAL’s “Virtual File Systems” to read remote files. GDAL has a lot of environment variables that control it’s behavior. Changing these settings can mean the difference being able to access a file or not. They can also have an impact on the performance.\nrio_env = rio.Env(GDAL_DISABLE_READDIR_ON_OPEN='TRUE',\n GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/cookies.txt'),\n GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/cookies.txt'))\nrio_env.__enter__()\n\n\nSet up rgdal configurations to access the cloud assets that we are interested in. You can learn more about these configuration options here.\nsetGDALconfig(c(\"GDAL_HTTP_UNSAFESSL=YES\",\n \"GDAL_HTTP_COOKIEFILE=.rcookies\",\n \"GDAL_HTTP_COOKIEJAR=.rcookies\",\n \"GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR\",\n \"CPL_VSIL_CURL_ALLOWED_EXTENSIONS=TIF\"))\n\n\n\nIn this example we’re interested in the HLS L30 data collection from NASA’s LP DAAC in Earthdata Cloud. Below we specify the HTTPS URL to the data asset in Earthdata Cloud. This URL can be found via Earthdata Search or programmatically through the CMR and CMR-STAC APIs.\n\nPythonR\n\n\nhttps_url = 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T11SQA.2021333T181532.v2.0/HLS.L30.T11SQA.2021333T181532.v2.0.B04.tif'\n\n\nPlease note that in R, we need to add /vsicurl/ manually to the COG file URL.\nhttps_url <- '/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T11SQA.2021333T181532.v2.0/HLS.L30.T11SQA.2021333T181532.v2.0.B04.tif'" + "objectID": "tutorials/Sea_Level_Rise.html#access-data", + "href": "tutorials/Sea_Level_Rise.html#access-data", + "title": "Analyzing Sea Level Rise with NASA Earthdata", + "section": "Access Data", + "text": "Access Data\nThe first dataset looks like it contains data from many altimetry missions, let’s explore a bit more! We will access the first granule of the SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205 collection in the month of May for every year that data is available and open the granules via xarray.\n\ngranules = []\nfor year in range(1990, 2019):\n granule = earthaccess.granule_query().short_name(\"SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205\").temporal(f\"{year}-05\", f\"{year}-06\").get(1)\n if len(granule)>0:\n granules.append(granule[0])\nprint(f\"Total granules: {len(granules)}\") \n\nTotal granules: 26\n\n\n\n%%time\n\nds_SSH = xr.open_mfdataset(earthaccess.open(granules), chunks={})\nds_SSH\n\n Opening 26 granules, approx size: 0.0 GB\nCPU times: user 5.79 s, sys: 846 ms, total: 6.64 s\nWall time: 24.9 s\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (Time: 26, Longitude: 2160, nv: 2, Latitude: 960)\nCoordinates:\n * Longitude (Longitude) float32 0.08333 0.25 0.4167 ... 359.6 359.8 359.9\n * Latitude (Latitude) float32 -79.92 -79.75 -79.58 ... 79.58 79.75 79.92\n * Time (Time) datetime64[ns] 1993-05-03T12:00:00 ... 2018-05-02T12:...\nDimensions without coordinates: nv\nData variables:\n Lon_bounds (Time, Longitude, nv) float32 dask.array<chunksize=(1, 2160, 2), meta=np.ndarray>\n Lat_bounds (Time, Latitude, nv) float32 dask.array<chunksize=(1, 960, 2), meta=np.ndarray>\n Time_bounds (Time, nv) datetime64[ns] dask.array<chunksize=(1, 2), meta=np.ndarray>\n SLA (Time, Latitude, Longitude) float32 dask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n SLA_ERR (Time, Latitude, Longitude) float32 dask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\nAttributes: (12/21)\n Conventions: CF-1.6\n ncei_template_version: NCEI_NetCDF_Grid_Template_v2.0\n Institution: Jet Propulsion Laboratory\n geospatial_lat_min: -79.916664\n geospatial_lat_max: 79.916664\n geospatial_lon_min: 0.083333336\n ... ...\n version_number: 2205\n Data_Pnts_Each_Sat: {\"16\": 780190, \"1001\": 668288}\n source_version: commit dc95db885c920084614a41849ce5a7d417198ef3\n SLA_Global_MEAN: -0.027657876081274502\n SLA_Global_STD: 0.0859016072308609\n latency: finalxarray.DatasetDimensions:Time: 26Longitude: 2160nv: 2Latitude: 960Coordinates: (3)Longitude(Longitude)float320.08333 0.25 0.4167 ... 359.8 359.9standard_name :longitudeunits :degrees_eastpoint_spacing :evenlong_name :longitudeaxis :Xbounds :Lon_boundsarray([8.333334e-02, 2.500000e-01, 4.166667e-01, ..., 3.595833e+02,\n 3.597500e+02, 3.599167e+02], dtype=float32)Latitude(Latitude)float32-79.92 -79.75 ... 79.75 79.92standard_name :latitudeunits :degrees_northpoint_spacing :evenlong_name :latitudeaxis :Ybounds :Lat_boundsarray([-79.916664, -79.75 , -79.583336, ..., 79.583336, 79.75 ,\n 79.916664], dtype=float32)Time(Time)datetime64[ns]1993-05-03T12:00:00 ... 2018-05-...standard_name :timelong_name :Timebounds :Time_boundsaxis :Tarray(['1993-05-03T12:00:00.000000000', '1994-05-03T12:00:00.000000000',\n '1995-05-03T12:00:00.000000000', '1996-05-02T12:00:00.000000000',\n '1997-05-02T12:00:00.000000000', '1998-05-02T12:00:00.000000000',\n '1999-05-02T12:00:00.000000000', '2000-05-01T12:00:00.000000000',\n '2001-05-01T12:00:00.000000000', '2002-05-01T12:00:00.000000000',\n '2003-05-01T12:00:00.000000000', '2004-05-05T12:00:00.000000000',\n '2005-05-05T12:00:00.000000000', '2006-05-05T12:00:00.000000000',\n '2007-05-05T12:00:00.000000000', '2008-05-04T12:00:00.000000000',\n '2009-05-04T12:00:00.000000000', '2010-05-04T12:00:00.000000000',\n '2011-05-04T12:00:00.000000000', '2012-05-03T12:00:00.000000000',\n '2013-05-03T12:00:00.000000000', '2014-05-03T12:00:00.000000000',\n '2015-05-03T12:00:00.000000000', '2016-05-02T12:00:00.000000000',\n '2017-05-02T12:00:00.000000000', '2018-05-02T12:00:00.000000000'],\n dtype='datetime64[ns]')Data variables: (5)Lon_bounds(Time, Longitude, nv)float32dask.array<chunksize=(1, 2160, 2), meta=np.ndarray>units :degrees_eastcomment :longitude values at the west and east bounds of each pixel.\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n438.75 kiB\n16.88 kiB\n\n\nShape\n(26, 2160, 2)\n(1, 2160, 2)\n\n\nDask graph\n26 chunks in 79 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\nLat_bounds\n\n\n(Time, Latitude, nv)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2), meta=np.ndarray>\n\n\n\n\nunits :\n\ndegrees_north\n\ncomment :\n\nlatitude values at the north and south bounds of each pixel.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n195.00 kiB\n7.50 kiB\n\n\nShape\n(26, 960, 2)\n(1, 960, 2)\n\n\nDask graph\n26 chunks in 79 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nTime_bounds\n\n\n(Time, nv)\n\n\ndatetime64[ns]\n\n\ndask.array<chunksize=(1, 2), meta=np.ndarray>\n\n\n\n\ncomment :\n\nTime bounds for each time value, same value as time variable. The time variable is defined on points instead of on bounding boxes.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n416 B\n16 B\n\n\nShape\n(26, 2)\n(1, 2)\n\n\nDask graph\n26 chunks in 53 graph layers\n\n\nData type\ndatetime64[ns] numpy.ndarray\n\n\n\n\n\n\n\n\n\nSLA\n\n\n(Time, Latitude, Longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n\n\n\n\nunits :\n\nm\n\nlong_name :\n\nSea Level Anomaly Estimate\n\nstandard_name :\n\nsea_surface_height_above_sea_level\n\nalias :\n\nsea_surface_height_above_sea_level\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n205.66 MiB\n7.91 MiB\n\n\nShape\n(26, 960, 2160)\n(1, 960, 2160)\n\n\nDask graph\n26 chunks in 53 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nSLA_ERR\n\n\n(Time, Latitude, Longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n\n\n\n\nunits :\n\nm\n\nlong_name :\n\nSea Level Anomaly Error Estimate\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n205.66 MiB\n7.91 MiB\n\n\nShape\n(26, 960, 2160)\n(1, 960, 2160)\n\n\nDask graph\n26 chunks in 53 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\nIndexes: (3)LongitudePandasIndexPandasIndex(Index([0.0833333358168602, 0.25, 0.4166666567325592,\n 0.5833333134651184, 0.75, 0.9166666865348816,\n 1.0833333730697632, 1.25, 1.4166666269302368,\n 1.5833333730697632,\n ...\n 358.4166564941406, 358.5833435058594, 358.75,\n 358.9166564941406, 359.0833435058594, 359.25,\n 359.4166564941406, 359.5833435058594, 359.75,\n 359.9166564941406],\n dtype='float32', name='Longitude', length=2160))LatitudePandasIndexPandasIndex(Index([-79.91666412353516, -79.75, -79.58333587646484,\n -79.41666412353516, -79.25, -79.08333587646484,\n -78.91666412353516, -78.75, -78.58333587646484,\n -78.41666412353516,\n ...\n 78.41666412353516, 78.58333587646484, 78.75,\n 78.91666412353516, 79.08333587646484, 79.25,\n 79.41666412353516, 79.58333587646484, 79.75,\n 79.91666412353516],\n dtype='float32', name='Latitude', length=960))TimePandasIndexPandasIndex(DatetimeIndex(['1993-05-03 12:00:00', '1994-05-03 12:00:00',\n '1995-05-03 12:00:00', '1996-05-02 12:00:00',\n '1997-05-02 12:00:00', '1998-05-02 12:00:00',\n '1999-05-02 12:00:00', '2000-05-01 12:00:00',\n '2001-05-01 12:00:00', '2002-05-01 12:00:00',\n '2003-05-01 12:00:00', '2004-05-05 12:00:00',\n '2005-05-05 12:00:00', '2006-05-05 12:00:00',\n '2007-05-05 12:00:00', '2008-05-04 12:00:00',\n '2009-05-04 12:00:00', '2010-05-04 12:00:00',\n '2011-05-04 12:00:00', '2012-05-03 12:00:00',\n '2013-05-03 12:00:00', '2014-05-03 12:00:00',\n '2015-05-03 12:00:00', '2016-05-02 12:00:00',\n '2017-05-02 12:00:00', '2018-05-02 12:00:00'],\n dtype='datetime64[ns]', name='Time', freq=None))Attributes: (21)Conventions :CF-1.6ncei_template_version :NCEI_NetCDF_Grid_Template_v2.0Institution :Jet Propulsion Laboratorygeospatial_lat_min :-79.916664geospatial_lat_max :79.916664geospatial_lon_min :0.083333336geospatial_lon_max :359.91666time_coverage_start :1993-05-03time_coverage_end :1993-05-03date_created :2022-10-30T20:40:50.716092title :Sea Level Anomaly Estimate based on Altimeter Data, final product (replaced interim version).short_name :SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205long_name :MEaSUREs Gridded Sea Surface Height Anomalies Version 2205summary :Sea level anomaly grids from altimeter data using Kriging interpolation, which gives best linear prediction based upon prior knowledge of covariance. DOI :10.5067/SLREF-CDRV3version_number :2205Data_Pnts_Each_Sat :{\"16\": 780190, \"1001\": 668288}source_version :commit dc95db885c920084614a41849ce5a7d417198ef3SLA_Global_MEAN :-0.027657876081274502SLA_Global_STD :0.0859016072308609latency :final" }, { - "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#https-data-access", - "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#https-data-access", - "title": "Accessing Cloud Optimized GeoTIFF (COG) - HTTPS Example", - "section": "HTTPS Data Access", - "text": "HTTPS Data Access\nRead in the HLS HTTPS URL for the L30 red band (0.64 – 0.67 μm) into our workspace. Note that, accessing files in the cloud requires you to authenticate using your NASA Earthdata Login account meaning a proper netrc file needs to be set up.\n\nPythonR\n\n\nwe are using rioxarray, an extension of xarray used to read geospatial data.\nda = rioxarray.open_rasterio(https_url)\nda\nThe file is read into Python as an xarray dataarray with a band, x, and y dimension. In this example the band dimension is meaningless, so we’ll use the squeeze() function to remove band as a dimension.\nda_red = da.squeeze('band', drop=True)\nda_red\nPlot the dataarray, representing the L30 red band, using hvplot.\nda_red.hvplot.image(x='x', y='y', cmap='gray', aspect='equal')\nExit the context manager.\nrio_env.__exit__()\n\n\nda_red <- rast(https_url)\nda_red\nThe Convert a SpatRaster object to a Raster object using raster() to be able to use leaflet to plot our data.\nred_raster <- da_red %>% raster()\nred_raster\nThen plot the red band using plot function.\nplot(red_raster)" + "objectID": "tutorials/Sea_Level_Rise.html#plot-sea-level-anomalies", + "href": "tutorials/Sea_Level_Rise.html#plot-sea-level-anomalies", + "title": "Analyzing Sea Level Rise with NASA Earthdata", + "section": "Plot Sea Level Anomalies", + "text": "Plot Sea Level Anomalies\n\nds_SSH.SLA.hvplot.image(y='Latitude', x='Longitude', cmap='viridis')" }, { - "objectID": "leading-workshops/add-folks-to-2i2c-github-teams.html", - "href": "leading-workshops/add-folks-to-2i2c-github-teams.html", - "title": "How to Add Folks to the 2i2c Hub", + "objectID": "tutorials/Sea_Level_Rise.html#recreate-the-sea-level-rise-infographic", + "href": "tutorials/Sea_Level_Rise.html#recreate-the-sea-level-rise-infographic", + "title": "Analyzing Sea Level Rise with NASA Earthdata", + "section": "Recreate the Sea Level Rise Infographic", + "text": "Recreate the Sea Level Rise Infographic\nFirst, we define a function that will calculate the area in 1/6 by 1/6 degree boxes in order to calculate the global mean of the SSH later.\n\ndef ssl_area(lats):\n \"\"\"\n Calculate the area associated with a 1/6 by 1/6 degree box at latitude specified in 'lats'.\n \n Parameter\n ==========\n lats: a list or numpy array of size N the latitudes of interest. \n \n Return\n =======\n out: Array (N) area values (unit: m^2)\n \"\"\"\n # Define WGS84 as CRS:\n geod = Geod(ellps='WGS84')\n dx=1/12.0\n # create a lambda function for calculating the perimeters of the boxes\n c_area=lambda lat: geod.polygon_area_perimeter(np.r_[-dx,dx,dx,-dx], lat+np.r_[-dx,-dx,dx,dx])[0]\n out=[]\n for lat in lats:\n out.append(c_area(lat))\n return np.array(out)\n\nLet’s use the function on our Sea Surface Height dataset.\n\n# note: they rotated the data in the last release, this operation used to be (1,-1)\nssh_area = ssl_area(ds_SSH.Latitude.data).reshape(-1,1)\nprint(ssh_area.shape)\n\n(960, 1)\n\n\nNext, we find and open the historic record dataset also using earthaccess and xarray.\n\nhistoric_ts_results = earthaccess.search_data(short_name='JPL_RECON_GMSL')\n\nGranules found: 1\n\n\n\nhistoric_ts=xr.open_mfdataset(earthaccess.open([historic_ts_results[0]]), engine='h5netcdf')\nhistoric_ts\n\n Opening 1 granules, approx size: 0.0 GB\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (time: 119)\nCoordinates:\n * time (time) datetime64[ns] ...\nData variables: (12/21)\n global_average_sea_level_change (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n global_average_sea_level_change_upper (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n global_average_sea_level_change_lower (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n glac_mean (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n glac_upper (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n glac_lower (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n ... ...\n global_average_thermosteric_sea_level_change (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n global_average_thermosteric_sea_level_change_upper (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n global_average_thermosteric_sea_level_change_lower (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n sum_of_contrib_processes_mean (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n sum_of_contrib_processes_upper (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n sum_of_contrib_processes_lower (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\nAttributes: (12/42)\n title: Global sea-level changes and contributing proc...\n summary: This file contains reconstructed global-mean s...\n id: 10.5067/GMSLT-FJPL1\n naming_authority: gov.nasa.jpl\n source: Frederikse et al. The causes of sea-level rise...\n project: NASA sea-level change science team (N-SLCT)\n ... ...\n time_coverage_start: 1900-01-01\n time_coverage_end: 2018-12-31\n time_coverage_duration: P119Y\n time_coverage_resolution: P1Y\n date_created: 2020-07-28\n date_modified: 2020-09-14xarray.DatasetDimensions:time: 119Coordinates: (1)time(time)datetime64[ns]1900-06-15 ... 2018-06-15long_name :timestandard_name :timearray(['1900-06-15T00:00:00.000000000', '1901-06-15T00:00:00.000000000',\n '1902-06-15T00:00:00.000000000', '1903-06-15T00:00:00.000000000',\n '1904-06-15T00:00:00.000000000', '1905-06-15T00:00:00.000000000',\n '1906-06-15T00:00:00.000000000', '1907-06-15T00:00:00.000000000',\n '1908-06-15T00:00:00.000000000', '1909-06-15T00:00:00.000000000',\n '1910-06-15T00:00:00.000000000', '1911-06-15T00:00:00.000000000',\n '1912-06-15T00:00:00.000000000', '1913-06-15T00:00:00.000000000',\n '1914-06-15T00:00:00.000000000', '1915-06-15T00:00:00.000000000',\n '1916-06-15T00:00:00.000000000', '1917-06-15T00:00:00.000000000',\n '1918-06-15T00:00:00.000000000', '1919-06-15T00:00:00.000000000',\n '1920-06-15T00:00:00.000000000', '1921-06-15T00:00:00.000000000',\n '1922-06-15T00:00:00.000000000', '1923-06-15T00:00:00.000000000',\n '1924-06-15T00:00:00.000000000', '1925-06-15T00:00:00.000000000',\n '1926-06-15T00:00:00.000000000', '1927-06-15T00:00:00.000000000',\n '1928-06-15T00:00:00.000000000', '1929-06-15T00:00:00.000000000',\n '1930-06-15T00:00:00.000000000', '1931-06-15T00:00:00.000000000',\n '1932-06-15T00:00:00.000000000', '1933-06-15T00:00:00.000000000',\n '1934-06-15T00:00:00.000000000', '1935-06-15T00:00:00.000000000',\n '1936-06-15T00:00:00.000000000', '1937-06-15T00:00:00.000000000',\n '1938-06-15T00:00:00.000000000', '1939-06-15T00:00:00.000000000',\n '1940-06-15T00:00:00.000000000', '1941-06-15T00:00:00.000000000',\n '1942-06-15T00:00:00.000000000', '1943-06-15T00:00:00.000000000',\n '1944-06-15T00:00:00.000000000', '1945-06-15T00:00:00.000000000',\n '1946-06-15T00:00:00.000000000', '1947-06-15T00:00:00.000000000',\n '1948-06-15T00:00:00.000000000', '1949-06-15T00:00:00.000000000',\n '1950-06-15T00:00:00.000000000', '1951-06-15T00:00:00.000000000',\n '1952-06-15T00:00:00.000000000', '1953-06-15T00:00:00.000000000',\n '1954-06-15T00:00:00.000000000', '1955-06-15T00:00:00.000000000',\n '1956-06-15T00:00:00.000000000', '1957-06-15T00:00:00.000000000',\n '1958-06-15T00:00:00.000000000', '1959-06-15T00:00:00.000000000',\n '1960-06-15T00:00:00.000000000', '1961-06-15T00:00:00.000000000',\n '1962-06-15T00:00:00.000000000', '1963-06-15T00:00:00.000000000',\n '1964-06-15T00:00:00.000000000', '1965-06-15T00:00:00.000000000',\n '1966-06-15T00:00:00.000000000', '1967-06-15T00:00:00.000000000',\n '1968-06-15T00:00:00.000000000', '1969-06-15T00:00:00.000000000',\n '1970-06-15T00:00:00.000000000', '1971-06-15T00:00:00.000000000',\n '1972-06-15T00:00:00.000000000', '1973-06-15T00:00:00.000000000',\n '1974-06-15T00:00:00.000000000', '1975-06-15T00:00:00.000000000',\n '1976-06-15T00:00:00.000000000', '1977-06-15T00:00:00.000000000',\n '1978-06-15T00:00:00.000000000', '1979-06-15T00:00:00.000000000',\n '1980-06-15T00:00:00.000000000', '1981-06-15T00:00:00.000000000',\n '1982-06-15T00:00:00.000000000', '1983-06-15T00:00:00.000000000',\n '1984-06-15T00:00:00.000000000', '1985-06-15T00:00:00.000000000',\n '1986-06-15T00:00:00.000000000', '1987-06-15T00:00:00.000000000',\n '1988-06-15T00:00:00.000000000', '1989-06-15T00:00:00.000000000',\n '1990-06-15T00:00:00.000000000', '1991-06-15T00:00:00.000000000',\n '1992-06-15T00:00:00.000000000', '1993-06-15T00:00:00.000000000',\n '1994-06-15T00:00:00.000000000', '1995-06-15T00:00:00.000000000',\n '1996-06-15T00:00:00.000000000', '1997-06-15T00:00:00.000000000',\n '1998-06-15T00:00:00.000000000', '1999-06-15T00:00:00.000000000',\n '2000-06-15T00:00:00.000000000', '2001-06-15T00:00:00.000000000',\n '2002-06-15T00:00:00.000000000', '2003-06-15T00:00:00.000000000',\n '2004-06-15T00:00:00.000000000', '2005-06-15T00:00:00.000000000',\n '2006-06-15T00:00:00.000000000', '2007-06-15T00:00:00.000000000',\n '2008-06-15T00:00:00.000000000', '2009-06-15T00:00:00.000000000',\n '2010-06-15T00:00:00.000000000', '2011-06-15T00:00:00.000000000',\n '2012-06-15T00:00:00.000000000', '2013-06-15T00:00:00.000000000',\n '2014-06-15T00:00:00.000000000', '2015-06-15T00:00:00.000000000',\n '2016-06-15T00:00:00.000000000', '2017-06-15T00:00:00.000000000',\n '2018-06-15T00:00:00.000000000'], dtype='datetime64[ns]')Data variables: (21)global_average_sea_level_change(time)float32dask.array<chunksize=(119,), meta=np.ndarray>units :mmlong_name :Observed global-average sea level (mean value)valid_min :-1000000.0valid_max :1000000.0comment :Value is relative to 2002-2019 baselinestandard_name :global_average_sea_level_changecoverage_content_type :physicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\nglobal_average_sea_level_change_upper\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nObserved global-average sea level (upper bound)\n\nstandard_name :\n\nglobal_average_sea_level_change\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nglobal_average_sea_level_change_lower\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nObserved global-mean sea level (lower bound)\n\nstandard_name :\n\nglobal_average_sea_level_change\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nglac_mean\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGlacier contribution (mean value)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. The glacier term does not include peripheral glaciers of both ice sheets\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nglac_upper\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGlacier contribution (upper bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. The glacier term does not include peripheral glaciers of both ice sheets\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nglac_lower\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGlacier contribution (lower bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. The glacier term does not include peripheral glaciers of both ice sheets\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nGrIS_mean\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGreenland Ice Sheet contribution (mean value)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. This term includes glaciers in the Greenland periphery\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nGrIS_upper\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGreenland Ice Sheet contribution (upper bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. This term includes glaciers in the Greenland periphery\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nGrIS_lower\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGreenland Ice Sheet contribution (lower bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. This term includes glaciers in the Greenland periphery\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nAIS_mean\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nAntarctic Ice Sheet contribution (mean value)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline.\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nAIS_upper\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nAntarctic Ice Sheet contribution (upper bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline.\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nAIS_lower\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nAntarctic Ice Sheet contribution (lower bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline.\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\ntws_mean\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nTerrestrial water storage contribution (mean value)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. This term includes the effects for groundwater depletion, reservoir impoundment and non-anthropogenic tws changes\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\ntws_upper\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nTerrestrial water storage contribution (upper bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. This term includes the effects for groundwater depletion, reservoir impoundment and non-anthropogenic tws changes\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\ntws_lower\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nTerrestrial water storage contribution (lower bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. This term includes the effects for groundwater depletion, reservoir impoundment and non-anthropogenic tws changes\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nglobal_average_thermosteric_sea_level_change\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGlobal thermosteric contribution (mean value)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline\n\nstandard_name :\n\nglobal_average_thermosteric_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nglobal_average_thermosteric_sea_level_change_upper\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGlobal thermosteric contribution (upper bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline\n\nstandard_name :\n\nglobal_average_thermosteric_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nglobal_average_thermosteric_sea_level_change_lower\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGlobal thermosteric contribution (lower bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline\n\nstandard_name :\n\nglobal_average_thermosteric_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nsum_of_contrib_processes_mean\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nSum of contributing processes (mean value)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nsum_of_contrib_processes_upper\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nSum of contributing processes (upper bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nsum_of_contrib_processes_lower\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nSum of contributing processes (lower bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nIndexes: (1)timePandasIndexPandasIndex(DatetimeIndex(['1900-06-15', '1901-06-15', '1902-06-15', '1903-06-15',\n '1904-06-15', '1905-06-15', '1906-06-15', '1907-06-15',\n '1908-06-15', '1909-06-15',\n ...\n '2009-06-15', '2010-06-15', '2011-06-15', '2012-06-15',\n '2013-06-15', '2014-06-15', '2015-06-15', '2016-06-15',\n '2017-06-15', '2018-06-15'],\n dtype='datetime64[ns]', name='time', length=119, freq=None))Attributes: (42)title :Global sea-level changes and contributing processes over 1900-2018summary :This file contains reconstructed global-mean sea level evolution and the estimated contributing processes over 1900-2018. Reconstructed sea level is based on annual-mean tide-gauge observations and uses the virtual-station method to aggregrate the individual observations into a global estimate. The contributing processes consist of thermosteric changes, glacier mass changes, mass changes of the Greenland and Antarctic Ice Sheet, and terrestrial water storage changes. The glacier, ice sheet, and terrestrial water storage are estimated by combining GRACE observations (2003-2018) with long-term estimates from in-situ observations and models. Steric estimates are based on in-situ temperature profiles. The upper- and lower bound represent the 5 and 95 percent confidence level. The numbers are equal to the ones presented in Frederikse et al. The causes of sea-level rise since 1900, Nature, 2020, reformatted to meet the specifications of the JPL PO.DAACid :10.5067/GMSLT-FJPL1naming_authority :gov.nasa.jplsource :Frederikse et al. The causes of sea-level rise since 1900, Nature, 2020 https://doi.org/10.1038/s41586-020-2591-3project :NASA sea-level change science team (N-SLCT)program :NASA sea-level change science team (N-SLCT)institution :NASA Jet Propulsion Laboratory (JPL)references :https://doi.org/10.5067/GMSLT-FJPL1,https://doi.org/10.1038/s41586-020-2591-3acknowledgement :This research was carried out by the Jet Propulsion Laboratory, managed by the California Institute of Technology under a contract with the National Aeronautics and Space Administration.processing_level :4product_version :1.0license :Public Domainhistory :This version provides the data as presented in Frederikse et al. 2020.Conventions :CF-1.7,ACDD-1.3keywords :EARTH SCIENCE > CLIMATE INDICATORS > ATMOSPHERIC/OCEAN INDICATORS > SEA LEVEL RISEkeywords_vocabulary :GCMD Science Keywordsstandard_name_vocabulary :CF Standard Name Table v27creator_name :Thomas Frederiksecreator_type :personcreator_url :https://sealevel.nasa.govcreator_email :thomas.frederikse@jpl.nasa.govcreator_institution :NASA Jet Propulsion Laboratory (JPL)contributor_name :Thomas Frederikse, Felix Landerer, Lambert Caron, Surendra Adhikari, David Parkes, Vincent Humphrey, Sönke Dangendorf, Peter Hogarth, Laure Zanna, Lijing Cheng, Yun-Hao Wucontributor_role :main author,author,author,author,author,author,author,author,author,author,authorpublisher_name :Physical Oceanography Distributed Active Archive Center (PO.DAAC)publisher_type :grouppublisher_url :https://podaac.jpl.nasa.govpublisher_email :podaac@podaac.jpl.nasa.govpublisher_institution :NASA Jet Propulsion Laboratory (JPL)geospatial_lat_min :-90.0geospatial_lat_max :90.0geospatial_lat_units :degrees_northgeospatial_lon_min :-180.0geospatial_lon_max :180.0geospatial_lon_units :degrees_easttime_coverage_start :1900-01-01time_coverage_end :2018-12-31time_coverage_duration :P119Ytime_coverage_resolution :P1Ydate_created :2020-07-28date_modified :2020-09-14\n\n\n\nLet’s Plot!\n\n%%time\n\nplt.rcParams[\"figure.figsize\"] = (16,4)\n\nfig, axs = plt.subplots()\nplt.grid(True)\n\n#function to get the global mean for plotting\ndef global_mean(SLA, **kwargs):\n dout=((SLA*ssh_area).sum()/(SLA/SLA*ssh_area).sum())*1000\n return dout\n\nresult = ds_SSH.SLA.groupby('Time').apply(global_mean)\n\nplt.xlabel('Time (year)',fontsize=16)\nplt.ylabel('Global Mean SLA (meter)',fontsize=12)\nplt.grid(True)\n\nresult.plot(ax=axs, color=\"orange\", marker=\"o\", label='Satellite Record')\n\nhistoric_ts['global_average_sea_level_change'].plot(ax=axs, label='Historical in-situ record', color=\"lightblue\")\n\nplt.legend()\nplt.show()\n\n\n\n\nCPU times: user 3.33 s, sys: 1.04 s, total: 4.37 s\nWall time: 4.05 s\n\n\n\nThis Data Story used content from Luis Lopez’s earthaccess tutorial, which was based on Jinbo Wang’s Earthdata Webinar tutorial." + }, + { + "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html", + "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html", + "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", "section": "", - "text": "We use GitHub Teams to manage access to the 2i2c Hub. To accomplish this, we use a dedicated GitHub Organization - nasa-openscapes-workshops - with GitHub Teams for admin (people who can add others to the Hub), workshop participants, Champions Cohorts, and long term access.\nPeople are given Openscapes 2i2c JupyterHub access for workshops run by NASA Openscapes Mentors. This is the workflow to add individual users one by one. When we have a list of workshop participants, there are quicker ways to add all at once; we’ll document that after testing." + "text": "This notebook demonstrates how to access OPeNDAP granules hosted inside of the Earthdata Cloud Archives. It shows how to query a cloud OPeNDAP-hosted Daymet granule using the Common Metadata Repository (CMR) API, before remotely accessing and analyzing it using Xarray and ncdump." }, { - "objectID": "leading-workshops/add-folks-to-2i2c-github-teams.html#share-google-form-with-participants", - "href": "leading-workshops/add-folks-to-2i2c-github-teams.html#share-google-form-with-participants", - "title": "How to Add Folks to the 2i2c Hub", - "section": "1. Share Google Form with participants", - "text": "1. Share Google Form with participants\nAsk participants: Please add your GitHub Username to this Google form for NASA Openscapes JupyterHub Access." + "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#import-packages", + "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#import-packages", + "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", + "section": "1. Import Packages", + "text": "1. Import Packages\n\nimport xarray as xr\nimport requests\nimport earthaccess\nimport datetime as dt\nimport pprint\nimport netCDF4 as nc4\nfrom subprocess import Popen\nimport platform\nimport os\nimport shutil" }, { - "objectID": "leading-workshops/add-folks-to-2i2c-github-teams.html#google-form-response-setup", - "href": "leading-workshops/add-folks-to-2i2c-github-teams.html#google-form-response-setup", - "title": "How to Add Folks to the 2i2c Hub", - "section": "2. Google Form response setup", - "text": "2. Google Form response setup\nSet the form responses Openscapes_JupyterHubAccess to notify when there are responses. Also do this for any separate sheets used for workshop JupyterHub registration Click on Responses > click three vertical dots and then check “email notification.”" + "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#create-edl-files-using-the-earthaccess-python-library", + "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#create-edl-files-using-the-earthaccess-python-library", + "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", + "section": "2. Create EDL files using the earthaccess Python library", + "text": "2. Create EDL files using the earthaccess Python library\nFirst, pass your Earthdata credentials to the earthaccess library to create the .netrc file:\n\nauth = earthaccess.login(strategy=\"interactive\", persist=True) \n\nYou're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 07/14/2023\nUsing user provided credentials for EDL\nPersisting credentials to .netrc\n\n\nNext, run the following code to generate the .dodsrc file, if it is not already present:\n\nhomeDir = os.path.expanduser(\"~\") + os.sep\n\nwith open(homeDir + '.dodsrc', 'w') as file:\n file.write('HTTP.COOKIEJAR={}.urs_cookies\\n'.format(homeDir))\n file.write('HTTP.NETRC={}.netrc'.format(homeDir))\n file.close()\n\nprint('Saved .dodsrc to:', homeDir)\n\n# Set appropriate permissions for Linux/macOS\nif platform.system() != \"Windows\":\n Popen('chmod og-rw ~/.netrc', shell=True)\nelse:\n # Copy dodsrc to working directory in Windows \n shutil.copy2(homeDir + '.dodsrc', os.getcwd())\n print('Copied .dodsrc to:', os.getcwd())" }, { - "objectID": "leading-workshops/add-folks-to-2i2c-github-teams.html#email-alert-of-new-signup", - "href": "leading-workshops/add-folks-to-2i2c-github-teams.html#email-alert-of-new-signup", - "title": "How to Add Folks to the 2i2c Hub", - "section": "3. Email alert of new signup", - "text": "3. Email alert of new signup\nWhen you get an email click ‘view response’ (not view summary)" + "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#finding-data-files-in-nasas-earthdata-cloud-or-whats-a-conceptid", + "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#finding-data-files-in-nasas-earthdata-cloud-or-whats-a-conceptid", + "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", + "section": "3. Finding Data Files in NASA’s EarthData Cloud: or What’s a ConceptID?", + "text": "3. Finding Data Files in NASA’s EarthData Cloud: or What’s a ConceptID?\nCloud OPeNDAP-enabled granules are organized b a unique identifier called the Concept ID. Below, we’ll demonstrate how to use NASA’s CMR API and either the Daymet dataset DOI or the shortname to obtain the Concept ID for Daymet Daily data. With this information, we can search all of NASA’s Earthdata holdings for matching data files (granules) based on our spatial area and time period of interest.\nSearch NASA Earthdata Holdings based on… * DOI - The Daymet daily data Digital Object Identifier * ConceptID - remember, this is NASA Earthdata’s unique ID for this dataset * Area of Interest * Time Range of Interest" }, { - "objectID": "leading-workshops/add-folks-to-2i2c-github-teams.html#copy-github-username", - "href": "leading-workshops/add-folks-to-2i2c-github-teams.html#copy-github-username", - "title": "How to Add Folks to the 2i2c Hub", - "section": "4. Copy GitHub username", - "text": "4. Copy GitHub username\nOn the response page, copy the Github Username" + "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#obtaining-the-daymet-v4-daily-data-concept-id", + "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#obtaining-the-daymet-v4-daily-data-concept-id", + "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", + "section": "3.1 Obtaining the Daymet V4 Daily data Concept ID", + "text": "3.1 Obtaining the Daymet V4 Daily data Concept ID\nUsing Python’s requests library, we will build and provide the URL doisearch to search NASA’s Common Metadata Repository (CMR) to acquire the concept_id unique to the Daymet V4 Daily data in NASA’s Cloud holdings. We’ll use the concept_id in a next step when we search NASA Cloud Holdings for our spatial and temporal area of interest.\n\n# CMR API base url\ncmrurl='https://cmr.earthdata.nasa.gov/search/' # define the base url of NASA's CMR API as the variable `cmrurl`\ndoi = '10.3334/ORNLDAAC/2129' # Daymet V4R1 DOI - define the DOI as the variable `daymet_doi`\ndoisearch = cmrurl + 'collections.json?doi=' + doi\nprint(doisearch)\n\nconcept_id = requests.get(doisearch).json()['feed']['entry'][0]['id']\nprint(concept_id)\n\nhttps://cmr.earthdata.nasa.gov/search/collections.json?doi=10.3334/ORNLDAAC/2129\nC2532426483-ORNL_CLOUD" }, { - "objectID": "leading-workshops/add-folks-to-2i2c-github-teams.html#add-github-username-to-a-github-team", - "href": "leading-workshops/add-folks-to-2i2c-github-teams.html#add-github-username-to-a-github-team", - "title": "How to Add Folks to the 2i2c Hub", - "section": "5. Add GitHub username to a GitHub Team", - "text": "5. Add GitHub username to a GitHub Team\nFor workshop participants, go to the WorkshopAccess-2i2c Team in GitHub > Click Add member > Paste GitHub username. Note there are a separate GitHub Teams for Champions Cohorts and long term access.\n\n\n\n\n\nGitHub will ask you for 2-factor authentication for the first one you add in a batch, not after that.\nPeople may see invitation notifications in two ways. They must accept the invitation to be able to access the Hub. This is what it looks like in their email inbox:\n\n\n\n\n\nThis is what the invitation looks like in the GitHub website interface - clicking the yellow banner notification leads to the invitation:" + "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#searching-for-daymet-files-using-the-cmr-search-api", + "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#searching-for-daymet-files-using-the-cmr-search-api", + "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", + "section": "4. Searching for Daymet Files using the CMR Search API", + "text": "4. Searching for Daymet Files using the CMR Search API\nDaymet daily data files (or granules) are in netCDF4 format, and each file has one year’s worth of data. Data files are organized by variables (each for dayl, prcp, tmin, tmax, srad, swe, vp) and regions (each for us, pr, hi). Daymet filenames can be used to identify the files from continental North America (*_na_*.nc). The files from Puerto Rico and Hawaii are named as (*_pr_*.nc) and (*_hi_*.nc) respectively.\nBelow, we create appropriately formatted strings for our temporal range:\n\nstart_date = dt.datetime(2010, 1, 1) # specify your own start date\nend_date = dt.datetime(2011, 12, 31) # specify your end start date\nprint(start_date,end_date,sep='\\n')\n\ndt_format = '%Y-%m-%dT%H:%M:%SZ' # format requirement for datetime search\ntemporal_str = start_date.strftime(dt_format) + ',' + end_date.strftime(dt_format)\nprint(temporal_str)\n\n2010-01-01 00:00:00\n2011-12-31 00:00:00\n2010-01-01T00:00:00Z,2011-12-31T00:00:00Z\n\n\nWe will first search all the granules for the full extent, and time period of interest (2010, 2011). For this tutorial, we set up the search parameters and download maximum temperature data (tmax).\n\ncmr_url = 'https://cmr.earthdata.nasa.gov/search/granules'\n\ncmr_response = requests.get(cmr_url, \n params={\n 'concept_id': concept_id,\n 'temporal': temporal_str,\n 'page_size': 200,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\n\ndaymet_granules = cmr_response.json()['feed']['entry']\n\ndaymet_granules_urls = []\n\nfor granule in daymet_granules:\n item = next((item['href'] for item in granule['links'] if \"opendap\" in item[\"href\"] and \"tmax\" in item[\"href\"]), None)\n if item != None:\n daymet_granules_urls.append(item)\n\ndaymet_granules_urls\n\n['https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_hi_tmax_2010.nc',\n 'https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_na_tmax_2010.nc',\n 'https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_pr_tmax_2010.nc',\n 'https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_hi_tmax_2011.nc',\n 'https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_pr_tmax_2011.nc',\n 'https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_na_tmax_2011.nc']" }, { - "objectID": "leading-workshops/add-folks-to-2i2c-github-teams.html#email-confirmation", - "href": "leading-workshops/add-folks-to-2i2c-github-teams.html#email-confirmation", - "title": "How to Add Folks to the 2i2c Hub", - "section": "6. Email confirmation", - "text": "6. Email confirmation\nGo back to the form and grab their email address. Send one of the following emails to the person.\nEmail for workshop specific: \n\nSubject: NASA Openscapes Access to 2i2c Hub\nHi [FIRST NAME],\nI have added you to the NASA Openscapes GitHub Organization’s Workshop Team. Please check your email and accept this invitation from GitHub. Once you have joined the team, you will be able to access the JupyterHub.\n- Here is the link to the hub: https://openscapes.2i2c.cloud/\n- Here is the material for the Workshop NAME> https://nasa-openscapes.github.io/2023-ssc/.\nHave a great workshop,\n\n…\nEmail for General requests: \n\nSubject: NASA Openscapes Access to 2i2c Hub Hi [FIRST NAME],\nI have added you to the NASA Openscapes 2i2c Jupyter Hub. Here is the link to the hub: https://openscapes.2i2c.cloud/ \nThere is a getting started guide in the NASA Earthdata Cloud Cookbook here: https://nasa-openscapes.github.io/earthdata-cloud-cookbook/\nWe’d love to know about the kind of work you are doing on the hub, and I’ll follow up in the next few months. \n\n\nBest,\n\n…" + "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#open-and-subset-granules-using-netcdf4-python-and-the-dap4-protocol", + "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#open-and-subset-granules-using-netcdf4-python-and-the-dap4-protocol", + "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", + "section": "5. Open and Subset Granules Using netcdf4-python and the DAP4 Protocol", + "text": "5. Open and Subset Granules Using netcdf4-python and the DAP4 Protocol\nThe “DAP4” protocol is included with Cloud OPeNDAP-enabled granules. This allows for certain granules to have their variables organized into group hierarchies, complex variable names retrieved, and to further distinguish dataset variables between each other. Because of this difference over on-premises OPeNDAP, which used DAP2, certain programming libraries may require updated methods for accessing Cloud OPeNDAP-enabled granules.\nIt is required to use the dap4:// protocol in the URL instead of https:// for the netcdf4-python library. Before the first granule in the list is accessed, we first replace the URL protocol in the string, then view its metadata using the netcdf4-python library.\nNOTE: Occasionally, due to server load, “BES Connection” errors may appear while opening a Cloud OPeNDAP granule. These errors do not affect granule access.\n\n#dap4_url = daymet_granules_urls.replace(\"https://\", \"dap4://\")\ndap4_url = [url.replace(\"https://\", \"dap4://\") for url in daymet_granules_urls]\n\nf = nc4.Dataset(dap4_url[0], 'r') # HI in 2010\nf\n\n<class 'netCDF4._netCDF4.Dataset'>\nroot group (NETCDF4 data model, file format DAP4):\n start_year: 2010\n source: Daymet Software Version 4.0\n Version_software: Daymet Software Version 4.0\n Version_data: Daymet Data Version 4.0\n Conventions: CF-1.6\n citation: Please see http://daymet.ornl.gov/ for current Daymet data citation information\n references: Please see http://daymet.ornl.gov/ for current information on Daymet references\n dimensions(sizes): y(584), x(284), time(365), nv(2)\n variables(dimensions): float32 y(y), float32 lon(y, x), float32 lat(y, x), float32 time(time), float32 x(x), float32 tmax(time, y, x), int16 lambert_conformal_conic(), float32 time_bnds(time, nv), int16 yearday(time)\n groups:" }, { - "objectID": "leading-workshops/add-folks-to-2i2c-github-teams.html#register-date-in-form-response-sheet", - "href": "leading-workshops/add-folks-to-2i2c-github-teams.html#register-date-in-form-response-sheet", - "title": "How to Add Folks to the 2i2c Hub", - "section": "7. Register date in Form Response Sheet", - "text": "7. Register date in Form Response Sheet\nIn the Form Response Sheet, in Column G add the date they were added to the Hub - in part so that we as admins knew if someone had already fulfilled the request. Rows 453+ are the most recent, but it can vary. Check Column A for the date of the form request." + "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#open-and-subset-granules-using-xarray", + "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#open-and-subset-granules-using-xarray", + "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", + "section": "6. Open and Subset Granules Using Xarray", + "text": "6. Open and Subset Granules Using Xarray\nXarray is a commonly-used and widely supported Python library used for accessing and analyzing remotely-hosted datasets. Below, we use the open_dataset() function to access our first Cloud OPeNDAP Daymet granule, and to view its metadata. Both the dap4:// and https:// protocols may be passed into the Xarray library, and subsetting must be done using Xarray.\n\nds = xr.open_dataset(daymet_granules_urls[0]) # HI in 2010\nds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (y: 584, x: 284, time: 365, nv: 2)\nCoordinates:\n * y (y) float32 -3.9e+04 -4e+04 ... -6.21e+05 -6.22e+05\n lon (y, x) float32 ...\n lat (y, x) float32 ...\n * time (time) datetime64[ns] 2010-01-01T12:00:00 ... 20...\n * x (x) float32 -5.802e+06 -5.801e+06 ... -5.519e+06\nDimensions without coordinates: nv\nData variables:\n tmax (time, y, x) float32 ...\n lambert_conformal_conic int16 -32767\n time_bnds (time, nv) datetime64[ns] 2010-01-01 ... 2011-01-01\n yearday (time) int16 1 2 3 4 5 6 ... 361 362 363 364 365\nAttributes:\n start_year: 2010\n source: Daymet Software Version 4.0\n Version_software: Daymet Software Version 4.0\n Version_data: Daymet Data Version 4.0\n Conventions: CF-1.6\n citation: Please see http://daymet.ornl.gov/ f...\n references: Please see http://daymet.ornl.gov/ f...\n build_dmrpp_metadata.build_dmrpp: 3.20.13-563\n build_dmrpp_metadata.bes: 3.20.13-563\n build_dmrpp_metadata.libdap: libdap-3.20.11-193\n build_dmrpp_metadata.configuration: \\n# TheBESKeys::get_as_config()\\nAll...\n build_dmrpp_metadata.invocation: build_dmrpp -c /tmp/bes_conf_p6Fo -f...xarray.DatasetDimensions:y: 584x: 284time: 365nv: 2Coordinates: (5)y(y)float32-3.9e+04 -4e+04 ... -6.22e+05units :mlong_name :y coordinate of projectionstandard_name :projection_y_coordinatearray([ -39000., -40000., -41000., ..., -620000., -621000., -622000.],\n dtype=float32)lon(y, x)float32...units :degrees_eastlong_name :longitude coordinatestandard_name :longitude[165856 values with dtype=float32]lat(y, x)float32...units :degrees_northlong_name :latitude coordinatestandard_name :latitude[165856 values with dtype=float32]time(time)datetime64[ns]2010-01-01T12:00:00 ... 2010-12-...standard_name :timebounds :time_bndslong_name :24-hour day based on local timearray(['2010-01-01T12:00:00.000000000', '2010-01-02T12:00:00.000000000',\n '2010-01-03T12:00:00.000000000', ..., '2010-12-29T12:00:00.000000000',\n '2010-12-30T12:00:00.000000000', '2010-12-31T12:00:00.000000000'],\n dtype='datetime64[ns]')x(x)float32-5.802e+06 ... -5.519e+06units :mlong_name :x coordinate of projectionstandard_name :projection_x_coordinatearray([-5802250., -5801250., -5800250., ..., -5521250., -5520250., -5519250.],\n dtype=float32)Data variables: (4)tmax(time, y, x)float32...long_name :daily maximum temperatureunits :degrees Cgrid_mapping :lambert_conformal_coniccell_methods :area: mean time: maximum[60537440 values with dtype=float32]lambert_conformal_conic()int16...grid_mapping_name :lambert_conformal_coniclongitude_of_central_meridian :-100.0latitude_of_projection_origin :42.5false_easting :0.0false_northing :0.0standard_parallel :[25. 60.]semi_major_axis :6378137.0inverse_flattening :298.257223563array(-32767, dtype=int16)time_bnds(time, nv)datetime64[ns]...array([['2010-01-01T00:00:00.000000000', '2010-01-02T00:00:00.000000000'],\n ['2010-01-02T00:00:00.000000000', '2010-01-03T00:00:00.000000000'],\n ['2010-01-03T00:00:00.000000000', '2010-01-04T00:00:00.000000000'],\n ...,\n ['2010-12-29T00:00:00.000000000', '2010-12-30T00:00:00.000000000'],\n ['2010-12-30T00:00:00.000000000', '2010-12-31T00:00:00.000000000'],\n ['2010-12-31T00:00:00.000000000', '2011-01-01T00:00:00.000000000']],\n dtype='datetime64[ns]')yearday(time)int16...long_name :day of year (DOY) starting with day 1 on Januaray 1starray([ 1, 2, 3, ..., 363, 364, 365], dtype=int16)Attributes: (12)start_year :2010source :Daymet Software Version 4.0Version_software :Daymet Software Version 4.0Version_data :Daymet Data Version 4.0Conventions :CF-1.6citation :Please see http://daymet.ornl.gov/ for current Daymet data citation informationreferences :Please see http://daymet.ornl.gov/ for current information on Daymet referencesbuild_dmrpp_metadata.build_dmrpp :3.20.13-563build_dmrpp_metadata.bes :3.20.13-563build_dmrpp_metadata.libdap :libdap-3.20.11-193build_dmrpp_metadata.configuration :\n# TheBESKeys::get_as_config()\nAllowedHosts=^https?:\\/\\/\nBES.Catalog.catalog.FollowSymLinks=Yes\nBES.Catalog.catalog.RootDirectory=/tmp/tmp9fn76qqj/\nBES.Catalog.catalog.TypeMatch=dmrpp:.*\\.(dmrpp)$;\nBES.Catalog.catalog.TypeMatch+=h5:.*(\\.bz2|\\.gz|\\.Z)?$;\nBES.Data.RootDirectory=/dev/null\nBES.LogName=./bes.log\nBES.UncompressCache.dir=/tmp/hyrax_ux\nBES.UncompressCache.prefix=ux_\nBES.UncompressCache.size=500\nBES.module.cmd=/usr/lib64/bes/libdap_xml_module.so\nBES.module.dap=/usr/lib64/bes/libdap_module.so\nBES.module.dmrpp=/usr/lib64/bes/libdmrpp_module.so\nBES.module.fonc=/usr/lib64/bes/libfonc_module.so\nBES.module.h5=/usr/lib64/bes/libhdf5_module.so\nBES.module.nc=/usr/lib64/bes/libnc_module.so\nBES.modules=dap,cmd,h5,dmrpp,nc,fonc\nFONc.ClassicModel=false\nFONc.NoGlobalAttrs=true\nH5.EnableCF=false\nH5.EnableCheckNameClashing=true\nbuild_dmrpp_metadata.invocation :build_dmrpp -c /tmp/bes_conf_p6Fo -f /tmp/tmp9fn76qqj//daymet_v4_daily_hi_tmax_2010.nc -r /tmp/dmr__CDJLKz -u OPeNDAP_DMRpp_DATA_ACCESS_URL -M" }, { - "objectID": "leading-workshops/add-folks-to-2i2c-github-teams.html#directions-for-workshop-participants", - "href": "leading-workshops/add-folks-to-2i2c-github-teams.html#directions-for-workshop-participants", - "title": "How to Add Folks to the 2i2c Hub", - "section": "Directions for workshop participants", - "text": "Directions for workshop participants\n\nPlease go to https://github.com/nasa-openscapes-workshops\nClick the big yellow bar that says “you’ve been invited” \nNot seeing that bar? \n\nAre you logged into GitHub.com?\nHave you shared your GitHub username via this form?\nYou can also check your email that you use for GitHub and look for an invitation from GitHub and NASA-Openscapes" + "objectID": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#resample-and-plot-tmax", + "href": "tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html#resample-and-plot-tmax", + "title": "Using OPeNDAP to Access Data from the Earthdata Cloud Archives", + "section": "7. Resample and Plot tmax", + "text": "7. Resample and Plot tmax\nBelow, we will resample the tmax variable and calculate the monthly mean using Xarray’s built-in functions. Then, we will plot the monthly tmax mean for the month of July in Hawaii for 2010.\n\n# Monthly resample\nmonthly_tmax_mean = ds['tmax'].resample(time=\"M\").mean()\nmonthly_tmax_mean\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray 'tmax' (time: 12, y: 584, x: 284)>\narray([[[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]],\n\n [[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]],\n\n [[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n...\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]],\n\n [[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]],\n\n [[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]]], dtype=float32)\nCoordinates:\n * time (time) datetime64[ns] 2010-01-31 2010-02-28 ... 2010-12-31\n * y (y) float32 -3.9e+04 -4e+04 -4.1e+04 ... -6.21e+05 -6.22e+05\n lon (y, x) float32 ...\n lat (y, x) float32 ...\n * x (x) float32 -5.802e+06 -5.801e+06 -5.8e+06 ... -5.52e+06 -5.519e+06xarray.DataArray'tmax'time: 12y: 584x: 284nan nan nan nan nan nan nan nan ... nan nan nan nan nan nan nan nanarray([[[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]],\n\n [[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]],\n\n [[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n...\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]],\n\n [[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]],\n\n [[nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n ...,\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan],\n [nan, nan, nan, ..., nan, nan, nan]]], dtype=float32)Coordinates: (5)time(time)datetime64[ns]2010-01-31 ... 2010-12-31array(['2010-01-31T00:00:00.000000000', '2010-02-28T00:00:00.000000000',\n '2010-03-31T00:00:00.000000000', '2010-04-30T00:00:00.000000000',\n '2010-05-31T00:00:00.000000000', '2010-06-30T00:00:00.000000000',\n '2010-07-31T00:00:00.000000000', '2010-08-31T00:00:00.000000000',\n '2010-09-30T00:00:00.000000000', '2010-10-31T00:00:00.000000000',\n '2010-11-30T00:00:00.000000000', '2010-12-31T00:00:00.000000000'],\n dtype='datetime64[ns]')y(y)float32-3.9e+04 -4e+04 ... -6.22e+05units :mlong_name :y coordinate of projectionstandard_name :projection_y_coordinatearray([ -39000., -40000., -41000., ..., -620000., -621000., -622000.],\n dtype=float32)lon(y, x)float32...units :degrees_eastlong_name :longitude coordinatestandard_name :longitude[165856 values with dtype=float32]lat(y, x)float32...units :degrees_northlong_name :latitude coordinatestandard_name :latitude[165856 values with dtype=float32]x(x)float32-5.802e+06 ... -5.519e+06units :mlong_name :x coordinate of projectionstandard_name :projection_x_coordinatearray([-5802250., -5801250., -5800250., ..., -5521250., -5520250., -5519250.],\n dtype=float32)Attributes: (0)\n\n\n\n# Xarray plotting\nmonthly_tmax_mean[6,:,:].plot()\n\n<matplotlib.collections.QuadMesh at 0x7fdaba88de50>" }, { - "objectID": "leading-workshops/add-folks-to-2i2c-github-teams.html#troubleshooting", - "href": "leading-workshops/add-folks-to-2i2c-github-teams.html#troubleshooting", - "title": "How to Add Folks to the 2i2c Hub", - "section": "Troubleshooting", - "text": "Troubleshooting\n\nUsers get 403 error\nThis is a common issue. User must find email notification from GitHub and accept invitation. If they had clicked in email to accept but did not complete accepting in GitHub, edit their invitation in GitHub to uninvite and reinvite. They should get new email to accept.\n\n\n\n\n\n\n\nDidn’t get email\nGo to https://github.com/nasa-openscapes-workshops, see the big yellow bar that says You’re invited." + "objectID": "tutorials/earthaccess-demo.html#why", + "href": "tutorials/earthaccess-demo.html#why", + "title": "Introducing NASA earthaccess 🌍", + "section": "Why?", + "text": "Why?\nProgrammatic, Easy, Reproducible.\nThere are many ways to access NASA datasets, we can use the Earthdata search portal. We can use DAAC specific portals or tools. We could even use data.gov! Web portals are great but they are not designed for programmatic access and reproducible workflows. This is extremely important in the age of the cloud and reproducible open science.\nThe good news is that NASA also exposes APIs that allows us to search, transform and access data in a programmatic way. Many of these libraries contain amazing features and some similarities. In this context, earthaccess aims to be a simple library that can deal with the important parts of the metadata so we can access or download data without having to worry if a given dataset is on-prem or in the cloud." }, { - "objectID": "leading-workshops/add-folks-to-2i2c-github-teams.html#notes-on-process-and-any-issues", - "href": "leading-workshops/add-folks-to-2i2c-github-teams.html#notes-on-process-and-any-issues", - "title": "How to Add Folks to the 2i2c Hub", - "section": "Notes on process and any issues", - "text": "Notes on process and any issues\nYuvi has started documenting the issues we are running into here -> https://hackmd.io/tWfqVai4SDC1CbQ4mhfIbw" + "objectID": "tutorials/earthaccess-demo.html#how", + "href": "tutorials/earthaccess-demo.html#how", + "title": "Introducing NASA earthaccess 🌍", + "section": "How?", + "text": "How?\n\nNote: There are a lot of acronyms that we need to get familiar with before any of this makes sense, here is a brief glossary for NASA Earthdata terms: NASA glossary\n\n\nAuthentication: Before we can use earthaccess we need an account with NASA EDL\nEarthdata Login provides free and immediate access to thousands of EOSDIS data products covering all Earth science disciplines and topic areas for researchers, applied science users, application developers, and the general public.\nOnce we have our NASA EDL login credentials we can start accessing NASA data in a programmatic way.\n\nimport earthaccess\nearthaccess.__version__\n\n'0.3.0'\n\n\n\nfrom earthaccess import Auth, Store, DataCollections, DataGranules\nauth = Auth()\n\n\nAuth()\nearthaccess’s Auth class provides 3 different strategies to authenticate ourselves with NASA EDL.\n\nnetrc: Do we have a .netrc file with our EDL credentials? if so, we can use it with earthaccess. If we don’t have it and want to create one we can, earthaccess allows users to type their credentials and persist them into a .netrc file.\nenvironment: If we have our EDL credentials as environment variables\n\nEDL_USERNAME\nEDL_PASSWORD\n\ninteractive: We will be asked for our EDL credentials with optinal persistance to .netrc\n\nTo persist our credentials to a .netrc file we have to do the following:\nauth.login(strategy=\"interactive\", persist=True)\n\nauth.login(strategy=\"netrc\")\n# are we authenticated?\nprint(auth.authenticated)\n\nYou're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 07/24/2022\nTrue\n\n\n\na = auth.login(strategy=\"environment\")\n# are we authenticated?\nprint(auth.authenticated)\n\nWe are already authenticated with NASA EDL\nTrue" }, { - "objectID": "environment-setup/index.html", - "href": "environment-setup/index.html", - "title": "Cloud Environment Setup", - "section": "", - "text": "This is primarily for Cloud environments, not locally." + "objectID": "tutorials/earthaccess-demo.html#querying-for-datasets", + "href": "tutorials/earthaccess-demo.html#querying-for-datasets", + "title": "Introducing NASA earthaccess 🌍", + "section": "Querying for datasets", + "text": "Querying for datasets\nThe DataCollections class can query CMR for any collection (dataset) using all of CMR’s Query parameters and has built-in functions to extract useful information from the response.\n\n# The first step is to create a DataCollections query \nQuery = DataCollections()\n\n# Use chain methods to customize our query\nQuery.keyword('elevation').bounding_box(-134.7,58.9,-133.9,59.2).temporal(\"2020-01-01\",\"2020-02-01\")\n\nprint(f'Collections found: {Query.hits()}')\n\n# filtering what UMM fields to print, to see the full record we omit the fields filters\n# meta is always included as \ncollections = Query.fields(['ShortName','Version']).get(5)\n# Inspect some results printing just the ShortName and Abstract\ncollections[0:3]\n\nThe results from a DataCollections and DataGranules query are enhanced python dictionaries, this means that we can access all the keys and values like we usually do with Python dictionaries.\n\ncollections[0][\"umm\"][\"ShortName\"]\n\nThe DataCollections class returns python dictionaries with some handy methods.\ncollection.concept_id() # returns the concept-id, used to search for data granules\ncollection.abstract() # returns the abstract\ncollection.landing_page() # returns the landing page if present in the UMM fields\ncollection.get_data() # returns the portal where data can be accessed.\nThe same results can be obtained using the dict syntax:\ncollection[\"meta\"][\"concept-id\"] # concept-id\ncollection[\"umm\"][\"RelatedUrls\"] # URLs, with GET DATA, LANDING PAGE etc\n\n# We can now search for collections using a pythonic API client for CMR.\nQuery = DataCollections().daac(\"PODAAC\")\n\nprint(f'Collections found: {Query.hits()}')\ncollections = Query.fields(['ShortName']).get(10)\n# Printing the first collection\ncollections[0]\n\n\n# What if we want cloud collections\nQuery = DataCollections().daac(\"PODAAC\").cloud_hosted(True)\n\nprint(f'Collections found: {Query.hits()}')\ncollections = Query.fields(['ShortName']).get(10)\n# Printing 3 collections\ncollections[0]\n\n\n# Printing the concept-id for the first 10 collections\n[collection.concept_id() for collection in collections]" }, { - "objectID": "environment-setup/index.html#how-environments-work", - "href": "environment-setup/index.html#how-environments-work", - "title": "Cloud Environment Setup", - "section": "How environments work", - "text": "How environments work\nComing soon." + "objectID": "tutorials/earthaccess-demo.html#querying-for-data-files-granules", + "href": "tutorials/earthaccess-demo.html#querying-for-data-files-granules", + "title": "Introducing NASA earthaccess 🌍", + "section": "Querying for data files (granules)", + "text": "Querying for data files (granules)\nThe DataGranules class provides similar functionality as the collection class. To query for granules in a more reliable way concept-id would be the main key. You can search data granules using a short name but that could (more likely will) return different versions of the same data granules.\nIn this example we’re querying for 10 data grnaules from ICESat-2 ATL06 version 005 dataset.\n\nNote: Generally speaking we won’t need authenticated queries unless they are restricted datasets for early adopters.\n\n\n# We build our query\nfrom pprint import pprint\nQuery = DataGranules().short_name('ATL06').version(\"005\").bounding_box(-134.7,58.9,-133.9,59.2)\n# We get 5 metadata records\ngranules = Query.get(5)\ngranules" }, { - "objectID": "environment-setup/index.html#corn", - "href": "environment-setup/index.html#corn", - "title": "Cloud Environment Setup", - "section": "Corn", - "text": "Corn\nCorn is a base image that allows the provisioning of a multi-kernel Docker base image for JupyterHub deployments. corn uses the amazing Pangeo’s base image, installs all the environments it finds under ci/environments and makes them available as kernels in the base image so users can select which kernel to use depending on their needs. We’re able to update this environment leveraging GitHub Actions and deployment. Corn has streamlined a lot of environment settings and has been a big leap forward for our work in the Cloud — and you can use it too.\nCorn full information coming soon. In the meantime, see:\n\nThe why, what, and how of our NASA Openscapes cloud infrastructure: 2i2c JupyterHub and corn environment - an awesome blog post by Corn creator Luis Lopez, November 2022\nNASA Openscapes Cloud Infrastructure - awesome slides and (video) by Luis Lopez in October 2022\nCorn GitHub Repository README and Issues\n\n\n\n\nIntegration between Openscapes and 2i2c. We update the environment via GitHub Actions and Docker deployment." + "objectID": "tutorials/earthaccess-demo.html#pretty-printing-data-granules", + "href": "tutorials/earthaccess-demo.html#pretty-printing-data-granules", + "title": "Introducing NASA earthaccess 🌍", + "section": "Pretty printing data granules", + "text": "Pretty printing data granules\nSince we are in a notebook we can take advantage of it to see a more user friendly version of the granules with the built-in function display This will render browse image for the granule if available and eventually will have a similar representation as the one from the Earthdata search portal\n\n# printing 2 granules using display\n[display(granule) for granule in granules]\n\n\nSpatiotemporal queries\nOur granules and collection classes accept the same spatial and temporal arguments as CMR so we can search for granules that match spatiotemporal criteria.\n\nQuery = DataGranules().short_name(\"ATL06\").temporal(\"2020-03-01\", \"2020-03-30\").bounding_box(-134.7,58.9,-133.9,59.2).version(\"005\")\n# Always inspects the hits before retrieven the granule metadata, just because it's very verbose.\nprint(f\"Granules found: {Query.hits()}\")\n\n\n# Now we can print some info about these granules using the built-in methods\ngranules = Query.get(5)\ndata_links = [{'links': g.data_links(access=\"on_prem\"), 'size (MB):': g.size()} for g in granules]\ndata_links" }, { - "objectID": "environment-setup/index.html#setting-up-corn-locally", - "href": "environment-setup/index.html#setting-up-corn-locally", - "title": "Cloud Environment Setup", - "section": "Setting up corn locally", - "text": "Setting up corn locally\nSetting up corn involves two steps: (1) Downloading the environment.yml, and (2) Setting up the environment using a package manager ( e.g. Anaconda Navigator, mamba, conda, etc.)\n\nDownload corn environment.yml\n\nNavigate to the corn GitHub repository page that contains the environment.yml file: https://github.com/NASA-Openscapes/corn/blob/main/ci/environment.yml\nTo download, on the file page, click the Raw button.\nSave as environment.yml\n\n\n\nUsing Anaconda Navigator\n\nOpen Anaconda Navigator\nImport environment.yml file\nName your environment something like nasaopenscapes_env [your unique name for this environment]\nValidating\n\nmore coming soon\nTODO - from local machine how will you connect to AWS?\nCloud Primer for Amazon Web Services from NASA EOSDIS" + "objectID": "tutorials/earthaccess-demo.html#accessing-the-data", + "href": "tutorials/earthaccess-demo.html#accessing-the-data", + "title": "Introducing NASA earthaccess 🌍", + "section": "Accessing the data", + "text": "Accessing the data\nWith earthaccess a researcher can get the files regardless if they are on-prem or cloud based with the same API call, although an important consideration is that if we want to access data in the cloud (direct access) we must run the code in the cloud. This is because some S3 buckets are configured to only allow direct access (s3:// links) if the requester is in the same zone, us-west-2." }, { - "objectID": "policies-usage/index.html", - "href": "policies-usage/index.html", - "title": "Policies & Usage Costs", - "section": "", - "text": "Our current access policies are found here: https://github.com/NASA-Openscapes/2i2cAccessPolicies" + "objectID": "tutorials/earthaccess-demo.html#on-prem-access-daac-hosted-data", + "href": "tutorials/earthaccess-demo.html#on-prem-access-daac-hosted-data", + "title": "Introducing NASA earthaccess 🌍", + "section": "On-prem access: DAAC hosted data 📡", + "text": "On-prem access: DAAC hosted data 📡\nThe Store() class will allow us to download or access our data and needs to be instantiated with our auth instance.\n\nstore = Store(auth)\n\nFor this example we are going to use a PODAAC dataset SMAP_JPL_L3_SSS_CAP_8DAY-RUNNINGMEAN_V5 which we previously queried (see querying for datasets) and got the concept id: C1972955240-PODAAC\n\nQuery = DataGranules().concept_id(\"C1972955240-PODAAC\").bounding_box(-134.7,54.9,-100.9,69.2)\nprint(f\"Granule hits: {Query.hits()}\")\n# getting more than 6,000 metadata records for demo purposes is going to slow us down a bit so let's get only a few\ngranules = Query.get(10)\n# Does this granule belong to a cloud-based collection?\ngranules[0].cloud_hosted\n\n\nFinally! let’s get the data\nThe Store class accepts the results from a DataGranules() query or it can also accept a list of URLs for the data files. In the second case we’ll have to specify the DAAC since it cannot infer which credentials to use solely on the URL.\n\n%%time\nfiles = store.get(granules[0:4], \"./data/C1972955240-PODAAC/\")" }, { - "objectID": "policies-usage/index.html#nasa-openscapes-policies", - "href": "policies-usage/index.html#nasa-openscapes-policies", - "title": "Policies & Usage Costs", - "section": "", - "text": "Our current access policies are found here: https://github.com/NASA-Openscapes/2i2cAccessPolicies" + "objectID": "tutorials/earthaccess-demo.html#accessing-the-data-in-the-cloud", + "href": "tutorials/earthaccess-demo.html#accessing-the-data-in-the-cloud", + "title": "Introducing NASA earthaccess 🌍", + "section": "Accessing the data in the cloud ☁️", + "text": "Accessing the data in the cloud ☁️\n\nWith earthaccess a researcher can get the files regardless if they are on-prem or cloud based with the same API call, although an important consideration is that if we want to access data in the cloud we must run the code in the cloud. This is because some S3 buckets are configured to only allow direct access (s3:// links) if the requester is in the same zone, us-west-2.\nSame API, just a different place, in this case the concept-id for the same dataset is C2208422957-POCLOUD > Note: The concept-id changed even though is the same dataset.\n\n\nQuery = DataGranules().concept_id(\"C2208422957-POCLOUD\").bounding_box(-134.7,54.9,-100.9,69.2)\nprint(f\"Granule hits: {Query.hits()}\")\ncloud_granules = Query.get(10)\n# is this a cloud hosted data granule?\ncloud_granules[0].cloud_hosted\n\n\n# Let's pretty print this\ncloud_granules[0]\n\n\n%%time\n# If we get an error with direct_access=True, most likely is because\n# we are running this code outside the us-west-2 region.\ntry:\n files = store.get(cloud_granules[0:4], local_path=\"./data/demo-POCLOUD\")\nexcept Exception as e:\n print(f\"Error: {e}, we are probably not using this code in the Amazon cloud. Trying external links...\")\n # There is hope, even if we are not in the Amazon cloud we can still get the data\n files = store.get(cloud_granules[0:4], access=\"external\", local_path=\"./data/demo-POCLOUD\")" }, { - "objectID": "policies-usage/index.html#aws-cost-explorer", - "href": "policies-usage/index.html#aws-cost-explorer", - "title": "Policies & Usage Costs", - "section": "AWS Cost Explorer", - "text": "AWS Cost Explorer\nThis is a work in progress, currently with minimal steps and screenshots that we will augment.\nAWS Cost Explorer lets you examine how much your usage costs. When using Credits, your usage does not immediately show up. Select Charge type as “Usage” from the right menu.\n\n\n\nAWS Cost Explorer. Charge type == “Usage”" + "objectID": "tutorials/earthaccess-demo.html#cloud-access-part-ii-streaming-data", + "href": "tutorials/earthaccess-demo.html#cloud-access-part-ii-streaming-data", + "title": "Introducing NASA earthaccess 🌍", + "section": "☁️ Cloud Access Part II: streaming data", + "text": "☁️ Cloud Access Part II: streaming data\nBeing in the cloud allows us to stream data as if we were using it locally. Pairing gridded datasets on S3 and xarray isa very useful patter when we deal with a lot of data.\n\nRecommended read: Skip the download! Stream NASA data directly into Python objects\n\n\nimport warnings\nwarnings.filterwarnings('ignore')\nwarnings.simplefilter('ignore')\nimport xarray as xr\n# data_links\nhttps_links = []\ns3_links = []\n\nfs = store.get_s3fs_session('POCLOUD')\n\nfor granule in cloud_granules:\n https_links.extend(granule.data_links(access=\"on_prem\"))\n s3_links.extend(granule.data_links(access=\"direct\"))\ns3_links\n\n\n%%time\n\nimport xarray as xr\n\ntry:\n files = store.open(s3_links, provider=\"POCLOUD\")\n\n ds_L3 = xr.open_mfdataset(\n files,\n combine='nested',\n concat_dim='time',\n decode_cf=True,\n coords='minimal',\n chunks={'time': 1}\n )\n ds_L3\nexcept Exception as e:\n pass\n # print(e)" }, { - "objectID": "policies-usage/index.html#aws-budgeting-alerts", - "href": "policies-usage/index.html#aws-budgeting-alerts", - "title": "Policies & Usage Costs", - "section": "AWS Budgeting Alerts", - "text": "AWS Budgeting Alerts\nThis is a work in progress, currently with minimal steps and screenshots that we will augment.\nThere are two types of alerts we set up.\n\nBudgeting alerts\nWhen adding new Cloud credits to our AWS account, we also create a budget and alerts (received via email) as we spend our credits. These are some beginning notes (credit and thank you to Joe Kennedy!).\nCreate an annual budget of the total Credits left (you may need to calculate this if credits rolled over at the beginning of the calendar year). In the Budget menu, create a Budget. Then select Customize and Cost budget.\n\n\n\n\n\nExclude Credits and Refunds, include Discounts. You can elect to receive emails with a regular cadence: weekly, monthly.\nWe set these up at 50, 75, 90, 95% of the total budget, and we will receive emails at those percentages. The thinking is that we will need to request more credits starting at 50-75%, and then make sure we have them in hand by 90-95%.\n\n\nThreshold alerts\nWe can also set up email alerts at certain dollar amounts.\n\n\n\n\n\nWe receive emails when we spend $100, $200, $500 of our Credits, which show up in the system as $1000 intervals." + "objectID": "tutorials/earthaccess-demo.html#now-to-the-important-science", + "href": "tutorials/earthaccess-demo.html#now-to-the-important-science", + "title": "Introducing NASA earthaccess 🌍", + "section": "Now to the important science! 🚀", + "text": "Now to the important science! 🚀\n\nRecap\n\nfrom earthaccess import Auth, DataGranules, Store\n\n# first we authenticate with NASA EDL\nauth = Auth().login(strategy=\"netrc\")\n\n# Then we build a Query with spatiotemporal parameters\nGranuleQuery = DataGranules().concept_id(\"C1575731655-LPDAAC_ECS\").bounding_box(-134.7,58.9,-133.9,59.2)\n\n# We get the metadata records from CMR\ngranules = GranuleQuery.get()\n\n# Now it{s time to download (or open) our data granules list with get()\nfiles = Store(auth).get(granules, local_path='./data')\n\n# Now to the important science!\n\n\nRelated links\nGithub repository: https://github.com/nsidc/earthaccess\nCMR API documentation: https://cmr.earthaccess.nasa.gov/search/site/docs/search/api.html\nEDL API documentation: https://urs.earthaccess.nasa.gov/\nNASA OpenScapes: https://nasa-openscapes.github.io/earthaccess-cloud-cookbook/\nNSIDC: https://nsidc.org\nContact: luis.lopez@nsidc.org" }, { - "objectID": "how-tos/using-s3-storage.html", - "href": "how-tos/using-s3-storage.html", - "title": "Using S3 Bucket Storage in NASA-Openscapes Hub", + "objectID": "tutorials/Above_Ground_Biomass_Density.html", + "href": "tutorials/Above_Ground_Biomass_Density.html", + "title": "Distribution of carbon stocks in a protected area", "section": "", - "text": "When you are working in the NASA Openscapes Hub, there are strategies we can use to manage our storage both in terms of cost and performance. The default storage location is the HOME directory (/home/jovyan/) mounted to the compute instance (the cloud computer that is doing the computations). The Hub uses an EC2 compute instance, with the HOME directory mounted to AWS Elastic File System (EFS) storage. This drive is really handy because it is persistent across server restarts and is a great place to store your code. However the HOME directory is not a great place to store data, as it is very expensive, and can also be quite slow to read from and write to.\nTo that end, the hub provides every user access to two AWS S3 buckets - a “scratch” bucket for short-term storage, and a “persistent” bucket for longer-term storage. S3 buckets have fast read/write, and storage costs are relatively inexpensive compared to storing in your HOME directory. A useful way to think of S3 buckets in relation to your compute instance is like attaching a cheap but fast external hard drive to your expensive laptop.\nOne other thing to note about these buckets is that all hub users can access each other’s user directories. These buckets are accessible only when you are working inside the hub; you can access them using the environment variables:\n\n$SCRATCH_BUCKET pointing to s3://openscapeshub-scratch/[your-username]\n\nScratch buckets are designed for storage of temporary files, e.g. intermediate results. Objects stored in a scratch bucket are removed after 7 days from their creation.\n\n$PERSISTENT_BUCKET pointing to s3://openscapeshub-persistent/[your-username]\n\nPersistent buckets are designed for storing data that is consistently used throughout the lifetime of a project. There is no automatic purging of objects in persistent buckets, so it is the responsibility of the hub admin and/or hub users to delete objects when they are no longer needed to minimize cloud billing costs.\n\n\nWe can interact with these directories in Python using the packages boto3 and/or s3fs, or in a terminal with the awsv2 cli tool. This tutorial will focus on using the s3fs package. See this page for more information on using S3 buckets in a 2i2c hub, and tips on using the aws cli tool.\n\n\nWe will start by accessing the same data we did in the Earthdata Cloud Clinic - reading it into memory as an xarray object and subsetting it.\n\nimport earthaccess \nimport xarray as xr\nimport hvplot.xarray #plot\nimport os\nimport tempfile\nimport s3fs # aws s3 access\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\nauth = earthaccess.login()\n\n\ndata_name = \"SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205\"\n\nresults = earthaccess.search_data(\n short_name=data_name,\n cloud_hosted=True,\n temporal=(\"2021-07-01\", \"2021-09-30\"),\n)\n\nGranules found: 18\n\n\n\nds = xr.open_mfdataset(earthaccess.open(results))\nds\n\nOpening 18 granules, approx size: 0.16 GB\nusing endpoint: https://archive.podaac.earthdata.nasa.gov/s3credentials\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset> Size: 299MB\nDimensions: (Time: 18, Longitude: 2160, nv: 2, Latitude: 960)\nCoordinates:\n * Longitude (Longitude) float32 9kB 0.08333 0.25 0.4167 ... 359.8 359.9\n * Latitude (Latitude) float32 4kB -79.92 -79.75 -79.58 ... 79.75 79.92\n * Time (Time) datetime64[ns] 144B 2021-07-05T12:00:00 ... 2021-09-2...\nDimensions without coordinates: nv\nData variables:\n Lon_bounds (Time, Longitude, nv) float32 311kB dask.array<chunksize=(1, 2160, 2), meta=np.ndarray>\n Lat_bounds (Time, Latitude, nv) float32 138kB dask.array<chunksize=(1, 960, 2), meta=np.ndarray>\n Time_bounds (Time, nv) datetime64[ns] 288B dask.array<chunksize=(1, 2), meta=np.ndarray>\n SLA (Time, Latitude, Longitude) float32 149MB dask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n SLA_ERR (Time, Latitude, Longitude) float32 149MB dask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\nAttributes: (12/21)\n Conventions: CF-1.6\n ncei_template_version: NCEI_NetCDF_Grid_Template_v2.0\n Institution: Jet Propulsion Laboratory\n geospatial_lat_min: -79.916664\n geospatial_lat_max: 79.916664\n geospatial_lon_min: 0.083333336\n ... ...\n version_number: 2205\n Data_Pnts_Each_Sat: {\"16\": 743215, \"1007\": 674076}\n source_version: commit 58c7da13c0c0069ae940c33a82bf1544b7d991bf\n SLA_Global_MEAN: 0.06428374482174487\n SLA_Global_STD: 0.0905195660534004\n latency: finalxarray.DatasetDimensions:Time: 18Longitude: 2160nv: 2Latitude: 960Coordinates: (3)Longitude(Longitude)float320.08333 0.25 0.4167 ... 359.8 359.9standard_name :longitudeunits :degrees_eastpoint_spacing :evenlong_name :longitudeaxis :Xbounds :Lon_boundsarray([8.333334e-02, 2.500000e-01, 4.166667e-01, ..., 3.595833e+02,\n 3.597500e+02, 3.599167e+02], dtype=float32)Latitude(Latitude)float32-79.92 -79.75 ... 79.75 79.92standard_name :latitudeunits :degrees_northpoint_spacing :evenlong_name :latitudeaxis :Ybounds :Lat_boundsarray([-79.916664, -79.75 , -79.583336, ..., 79.583336, 79.75 ,\n 79.916664], dtype=float32)Time(Time)datetime64[ns]2021-07-05T12:00:00 ... 2021-09-...standard_name :timelong_name :Timebounds :Time_boundsaxis :Tarray(['2021-07-05T12:00:00.000000000', '2021-07-10T12:00:00.000000000',\n '2021-07-15T12:00:00.000000000', '2021-07-20T12:00:00.000000000',\n '2021-07-25T12:00:00.000000000', '2021-07-30T12:00:00.000000000',\n '2021-08-04T12:00:00.000000000', '2021-08-09T12:00:00.000000000',\n '2021-08-14T12:00:00.000000000', '2021-08-19T12:00:00.000000000',\n '2021-08-24T12:00:00.000000000', '2021-08-29T12:00:00.000000000',\n '2021-09-03T12:00:00.000000000', '2021-09-08T12:00:00.000000000',\n '2021-09-13T12:00:00.000000000', '2021-09-18T12:00:00.000000000',\n '2021-09-23T12:00:00.000000000', '2021-09-28T12:00:00.000000000'],\n dtype='datetime64[ns]')Data variables: (5)Lon_bounds(Time, Longitude, nv)float32dask.array<chunksize=(1, 2160, 2), meta=np.ndarray>units :degrees_eastcomment :longitude values at the west and east bounds of each pixel.\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n303.75 kiB\n16.88 kiB\n\n\nShape\n(18, 2160, 2)\n(1, 2160, 2)\n\n\nDask graph\n18 chunks in 55 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\nLat_bounds\n\n\n(Time, Latitude, nv)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2), meta=np.ndarray>\n\n\n\n\nunits :\n\ndegrees_north\n\ncomment :\n\nlatitude values at the north and south bounds of each pixel.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n135.00 kiB\n7.50 kiB\n\n\nShape\n(18, 960, 2)\n(1, 960, 2)\n\n\nDask graph\n18 chunks in 55 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nTime_bounds\n\n\n(Time, nv)\n\n\ndatetime64[ns]\n\n\ndask.array<chunksize=(1, 2), meta=np.ndarray>\n\n\n\n\ncomment :\n\nTime bounds for each time value, same value as time variable. The time variable is defined on points instead of on bounding boxes.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n288 B\n16 B\n\n\nShape\n(18, 2)\n(1, 2)\n\n\nDask graph\n18 chunks in 37 graph layers\n\n\nData type\ndatetime64[ns] numpy.ndarray\n\n\n\n\n\n\n\n\n\nSLA\n\n\n(Time, Latitude, Longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n\n\n\n\nunits :\n\nm\n\nlong_name :\n\nSea Level Anomaly Estimate\n\nstandard_name :\n\nsea_surface_height_above_sea_level\n\nalias :\n\nsea_surface_height_above_sea_level\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n142.38 MiB\n7.91 MiB\n\n\nShape\n(18, 960, 2160)\n(1, 960, 2160)\n\n\nDask graph\n18 chunks in 37 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nSLA_ERR\n\n\n(Time, Latitude, Longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n\n\n\n\nunits :\n\nm\n\nlong_name :\n\nSea Level Anomaly Error Estimate\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n142.38 MiB\n7.91 MiB\n\n\nShape\n(18, 960, 2160)\n(1, 960, 2160)\n\n\nDask graph\n18 chunks in 37 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\nIndexes: (3)LongitudePandasIndexPandasIndex(Index([0.0833333358168602, 0.25, 0.4166666567325592,\n 0.5833333134651184, 0.75, 0.9166666865348816,\n 1.0833333730697632, 1.25, 1.4166666269302368,\n 1.5833333730697632,\n ...\n 358.4166564941406, 358.5833435058594, 358.75,\n 358.9166564941406, 359.0833435058594, 359.25,\n 359.4166564941406, 359.5833435058594, 359.75,\n 359.9166564941406],\n dtype='float32', name='Longitude', length=2160))LatitudePandasIndexPandasIndex(Index([-79.91666412353516, -79.75, -79.58333587646484,\n -79.41666412353516, -79.25, -79.08333587646484,\n -78.91666412353516, -78.75, -78.58333587646484,\n -78.41666412353516,\n ...\n 78.41666412353516, 78.58333587646484, 78.75,\n 78.91666412353516, 79.08333587646484, 79.25,\n 79.41666412353516, 79.58333587646484, 79.75,\n 79.91666412353516],\n dtype='float32', name='Latitude', length=960))TimePandasIndexPandasIndex(DatetimeIndex(['2021-07-05 12:00:00', '2021-07-10 12:00:00',\n '2021-07-15 12:00:00', '2021-07-20 12:00:00',\n '2021-07-25 12:00:00', '2021-07-30 12:00:00',\n '2021-08-04 12:00:00', '2021-08-09 12:00:00',\n '2021-08-14 12:00:00', '2021-08-19 12:00:00',\n '2021-08-24 12:00:00', '2021-08-29 12:00:00',\n '2021-09-03 12:00:00', '2021-09-08 12:00:00',\n '2021-09-13 12:00:00', '2021-09-18 12:00:00',\n '2021-09-23 12:00:00', '2021-09-28 12:00:00'],\n dtype='datetime64[ns]', name='Time', freq=None))Attributes: (21)Conventions :CF-1.6ncei_template_version :NCEI_NetCDF_Grid_Template_v2.0Institution :Jet Propulsion Laboratorygeospatial_lat_min :-79.916664geospatial_lat_max :79.916664geospatial_lon_min :0.083333336geospatial_lon_max :359.91666time_coverage_start :2021-07-05time_coverage_end :2021-07-05date_created :2022-10-30T20:58:16.297288title :Sea Level Anomaly Estimate based on Altimeter Data, final product (replaced interim version).short_name :SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205long_name :MEaSUREs Gridded Sea Surface Height Anomalies Version 2205summary :Sea level anomaly grids from altimeter data using Kriging interpolation, which gives best linear prediction based upon prior knowledge of covariance. DOI :10.5067/SLREF-CDRV3version_number :2205Data_Pnts_Each_Sat :{\"16\": 743215, \"1007\": 674076}source_version :commit 58c7da13c0c0069ae940c33a82bf1544b7d991bfSLA_Global_MEAN :0.06428374482174487SLA_Global_STD :0.0905195660534004latency :final\n\n\n\nds_subset = ds['SLA'].sel(Latitude=slice(15.8, 35.9), Longitude=slice(234.5,260.5)) \nds_subset\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray 'SLA' (Time: 18, Latitude: 120, Longitude: 156)> Size: 1MB\ndask.array<getitem, shape=(18, 120, 156), dtype=float32, chunksize=(1, 120, 156), chunktype=numpy.ndarray>\nCoordinates:\n * Longitude (Longitude) float32 624B 234.6 234.8 234.9 ... 260.1 260.2 260.4\n * Latitude (Latitude) float32 480B 15.92 16.08 16.25 ... 35.42 35.58 35.75\n * Time (Time) datetime64[ns] 144B 2021-07-05T12:00:00 ... 2021-09-28T...\nAttributes:\n units: m\n long_name: Sea Level Anomaly Estimate\n standard_name: sea_surface_height_above_sea_level\n alias: sea_surface_height_above_sea_levelxarray.DataArray'SLA'Time: 18Latitude: 120Longitude: 156dask.array<chunksize=(1, 120, 156), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n1.29 MiB\n73.12 kiB\n\n\nShape\n(18, 120, 156)\n(1, 120, 156)\n\n\nDask graph\n18 chunks in 38 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\nCoordinates: (3)Longitude(Longitude)float32234.6 234.8 234.9 ... 260.2 260.4standard_name :longitudeunits :degrees_eastpoint_spacing :evenlong_name :longitudeaxis :Xbounds :Lon_boundsarray([234.58333, 234.75 , 234.91667, 235.08333, 235.25 , 235.41667,\n 235.58333, 235.75 , 235.91667, 236.08333, 236.25 , 236.41667,\n 236.58333, 236.75 , 236.91667, 237.08333, 237.25 , 237.41667,\n 237.58333, 237.75 , 237.91667, 238.08333, 238.25 , 238.41667,\n 238.58333, 238.75 , 238.91667, 239.08333, 239.25 , 239.41667,\n 239.58333, 239.75 , 239.91667, 240.08333, 240.25 , 240.41667,\n 240.58333, 240.75 , 240.91667, 241.08333, 241.25 , 241.41667,\n 241.58333, 241.75 , 241.91667, 242.08333, 242.25 , 242.41667,\n 242.58333, 242.75 , 242.91667, 243.08333, 243.25 , 243.41667,\n 243.58333, 243.75 , 243.91667, 244.08333, 244.25 , 244.41667,\n 244.58333, 244.75 , 244.91667, 245.08333, 245.25 , 245.41667,\n 245.58333, 245.75 , 245.91667, 246.08333, 246.25 , 246.41667,\n 246.58333, 246.75 , 246.91667, 247.08333, 247.25 , 247.41667,\n 247.58333, 247.75 , 247.91667, 248.08333, 248.25 , 248.41667,\n 248.58333, 248.75 , 248.91667, 249.08333, 249.25 , 249.41667,\n 249.58333, 249.75 , 249.91667, 250.08333, 250.25 , 250.41667,\n 250.58333, 250.75 , 250.91667, 251.08333, 251.25 , 251.41667,\n 251.58333, 251.75 , 251.91667, 252.08333, 252.25 , 252.41667,\n 252.58333, 252.75 , 252.91667, 253.08333, 253.25 , 253.41667,\n 253.58333, 253.75 , 253.91667, 254.08333, 254.25 , 254.41667,\n 254.58333, 254.75 , 254.91667, 255.08333, 255.25 , 255.41667,\n 255.58333, 255.75 , 255.91667, 256.08334, 256.25 , 256.41666,\n 256.58334, 256.75 , 256.91666, 257.08334, 257.25 , 257.41666,\n 257.58334, 257.75 , 257.91666, 258.08334, 258.25 , 258.41666,\n 258.58334, 258.75 , 258.91666, 259.08334, 259.25 , 259.41666,\n 259.58334, 259.75 , 259.91666, 260.08334, 260.25 , 260.41666],\n dtype=float32)Latitude(Latitude)float3215.92 16.08 16.25 ... 35.58 35.75standard_name :latitudeunits :degrees_northpoint_spacing :evenlong_name :latitudeaxis :Ybounds :Lat_boundsarray([15.916667, 16.083334, 16.25 , 16.416666, 16.583334, 16.75 ,\n 16.916666, 17.083334, 17.25 , 17.416666, 17.583334, 17.75 ,\n 17.916666, 18.083334, 18.25 , 18.416666, 18.583334, 18.75 ,\n 18.916666, 19.083334, 19.25 , 19.416666, 19.583334, 19.75 ,\n 19.916666, 20.083334, 20.25 , 20.416666, 20.583334, 20.75 ,\n 20.916666, 21.083334, 21.25 , 21.416666, 21.583334, 21.75 ,\n 21.916666, 22.083334, 22.25 , 22.416666, 22.583334, 22.75 ,\n 22.916666, 23.083334, 23.25 , 23.416666, 23.583334, 23.75 ,\n 23.916666, 24.083334, 24.25 , 24.416666, 24.583334, 24.75 ,\n 24.916666, 25.083334, 25.25 , 25.416666, 25.583334, 25.75 ,\n 25.916666, 26.083334, 26.25 , 26.416666, 26.583334, 26.75 ,\n 26.916666, 27.083334, 27.25 , 27.416666, 27.583334, 27.75 ,\n 27.916666, 28.083334, 28.25 , 28.416666, 28.583334, 28.75 ,\n 28.916666, 29.083334, 29.25 , 29.416666, 29.583334, 29.75 ,\n 29.916666, 30.083334, 30.25 , 30.416666, 30.583334, 30.75 ,\n 30.916666, 31.083334, 31.25 , 31.416666, 31.583334, 31.75 ,\n 31.916666, 32.083332, 32.25 , 32.416668, 32.583332, 32.75 ,\n 32.916668, 33.083332, 33.25 , 33.416668, 33.583332, 33.75 ,\n 33.916668, 34.083332, 34.25 , 34.416668, 34.583332, 34.75 ,\n 34.916668, 35.083332, 35.25 , 35.416668, 35.583332, 35.75 ],\n dtype=float32)Time(Time)datetime64[ns]2021-07-05T12:00:00 ... 2021-09-...standard_name :timelong_name :Timebounds :Time_boundsaxis :Tarray(['2021-07-05T12:00:00.000000000', '2021-07-10T12:00:00.000000000',\n '2021-07-15T12:00:00.000000000', '2021-07-20T12:00:00.000000000',\n '2021-07-25T12:00:00.000000000', '2021-07-30T12:00:00.000000000',\n '2021-08-04T12:00:00.000000000', '2021-08-09T12:00:00.000000000',\n '2021-08-14T12:00:00.000000000', '2021-08-19T12:00:00.000000000',\n '2021-08-24T12:00:00.000000000', '2021-08-29T12:00:00.000000000',\n '2021-09-03T12:00:00.000000000', '2021-09-08T12:00:00.000000000',\n '2021-09-13T12:00:00.000000000', '2021-09-18T12:00:00.000000000',\n '2021-09-23T12:00:00.000000000', '2021-09-28T12:00:00.000000000'],\n dtype='datetime64[ns]')Indexes: (3)LongitudePandasIndexPandasIndex(Index([234.5833282470703, 234.75, 234.9166717529297,\n 235.0833282470703, 235.25, 235.4166717529297,\n 235.5833282470703, 235.75, 235.9166717529297,\n 236.0833282470703,\n ...\n 258.9166564941406, 259.0833435058594, 259.25,\n 259.4166564941406, 259.5833435058594, 259.75,\n 259.9166564941406, 260.0833435058594, 260.25,\n 260.4166564941406],\n dtype='float32', name='Longitude', length=156))LatitudePandasIndexPandasIndex(Index([15.916666984558105, 16.08333396911621, 16.25,\n 16.41666603088379, 16.58333396911621, 16.75,\n 16.91666603088379, 17.08333396911621, 17.25,\n 17.41666603088379,\n ...\n 34.25, 34.41666793823242, 34.58333206176758,\n 34.75, 34.91666793823242, 35.08333206176758,\n 35.25, 35.41666793823242, 35.58333206176758,\n 35.75],\n dtype='float32', name='Latitude', length=120))TimePandasIndexPandasIndex(DatetimeIndex(['2021-07-05 12:00:00', '2021-07-10 12:00:00',\n '2021-07-15 12:00:00', '2021-07-20 12:00:00',\n '2021-07-25 12:00:00', '2021-07-30 12:00:00',\n '2021-08-04 12:00:00', '2021-08-09 12:00:00',\n '2021-08-14 12:00:00', '2021-08-19 12:00:00',\n '2021-08-24 12:00:00', '2021-08-29 12:00:00',\n '2021-09-03 12:00:00', '2021-09-08 12:00:00',\n '2021-09-13 12:00:00', '2021-09-18 12:00:00',\n '2021-09-23 12:00:00', '2021-09-28 12:00:00'],\n dtype='datetime64[ns]', name='Time', freq=None))Attributes: (4)units :mlong_name :Sea Level Anomaly Estimatestandard_name :sea_surface_height_above_sea_levelalias :sea_surface_height_above_sea_level\n\n\n\n\n\nImagining this ds_subset object is now an important intermediate dataset, or the result of a complex analysis and we want to save it. Our default action might be to just save it to our HOME directory. This is simple, but we want to avoid this as it incurs significant storage costs, and using this data later will be slow.\nds_subset.to_netcdf(\"test.nc\") # avoid writing to home directory like this\n\n\n\ns3fs is a Python library that allows us to interact with S3 objects in a file-system like manner.\n\n# Create a S3FileSystem class\ns3 = s3fs.S3FileSystem()\n\n# Get scratch and persistent buckets\nscratch = os.environ[\"SCRATCH_BUCKET\"]\npersistent = os.environ[\"PERSISTENT_BUCKET\"]\n\nprint(scratch)\nprint(persistent)\n\ns3://openscapeshub-scratch/ateucher\ns3://openscapeshub-persistent/ateucher\n\n\nOur user-specific directories in the two buckets aren’t actually created until we put something in them, so if we try to check for their existence or list their contents before they are created, we will get an error. We will use the S3FileSystem.touch() method to place a simple empty file called .placeholder in each one to bring them into existence.\n\ns3.touch(f\"{scratch}/.placeholder\")\n\ns3.ls(scratch)\n\n['openscapeshub-scratch/ateucher/.placeholder']\n\n\nand in our persistent bucket:\n\ns3.touch(f\"{persistent}/.placeholder\")\n\ns3.ls(persistent)\n\n['openscapeshub-persistent/ateucher/.placeholder']\n\n\n(Note that adding these placeholders isn’t strictly necessary, as the first time you write anything to these buckets they will be created.)\n\n\n\nNext we can save ds_subset as a netcdf file in our scratch bucket. This involves writing to a temporary directory first, and then moving that to the SCRATCH bucket:\n\n# Where we want to store it:\nscratch_nc_file_path = f\"{scratch}/test123.nc\"\n\n# Create a temporary intermediate file and save it to the bucket\nwith tempfile.NamedTemporaryFile(suffix = \".nc\") as tmp:\n ds_subset.to_netcdf(tmp.name) # save it to a temporary file\n s3.put(tmp.name, scratch_nc_file_path) # move that file to the scratch bucket\n\n# Ensure the file is there\ns3.ls(scratch)\n\n['openscapeshub-scratch/ateucher/.placeholder',\n 'openscapeshub-scratch/ateucher/test123.nc']\n\n\nAnd we can open it to ensure it worked:\n\nds_from_scratch = xr.open_dataarray(s3.open(scratch_nc_file_path))\n\nds_from_scratch\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray 'SLA' (Time: 18, Latitude: 120, Longitude: 156)> Size: 1MB\n[336960 values with dtype=float32]\nCoordinates:\n * Longitude (Longitude) float32 624B 234.6 234.8 234.9 ... 260.1 260.2 260.4\n * Latitude (Latitude) float32 480B 15.92 16.08 16.25 ... 35.42 35.58 35.75\n * Time (Time) datetime64[ns] 144B 2021-07-05T12:00:00 ... 2021-09-28T...\nAttributes:\n units: m\n long_name: Sea Level Anomaly Estimate\n standard_name: sea_surface_height_above_sea_level\n alias: sea_surface_height_above_sea_levelxarray.DataArray'SLA'Time: 18Latitude: 120Longitude: 156...[336960 values with dtype=float32]Coordinates: (3)Longitude(Longitude)float32234.6 234.8 234.9 ... 260.2 260.4standard_name :longitudeunits :degrees_eastpoint_spacing :evenlong_name :longitudeaxis :Xbounds :Lon_boundsarray([234.58333, 234.75 , 234.91667, 235.08333, 235.25 , 235.41667,\n 235.58333, 235.75 , 235.91667, 236.08333, 236.25 , 236.41667,\n 236.58333, 236.75 , 236.91667, 237.08333, 237.25 , 237.41667,\n 237.58333, 237.75 , 237.91667, 238.08333, 238.25 , 238.41667,\n 238.58333, 238.75 , 238.91667, 239.08333, 239.25 , 239.41667,\n 239.58333, 239.75 , 239.91667, 240.08333, 240.25 , 240.41667,\n 240.58333, 240.75 , 240.91667, 241.08333, 241.25 , 241.41667,\n 241.58333, 241.75 , 241.91667, 242.08333, 242.25 , 242.41667,\n 242.58333, 242.75 , 242.91667, 243.08333, 243.25 , 243.41667,\n 243.58333, 243.75 , 243.91667, 244.08333, 244.25 , 244.41667,\n 244.58333, 244.75 , 244.91667, 245.08333, 245.25 , 245.41667,\n 245.58333, 245.75 , 245.91667, 246.08333, 246.25 , 246.41667,\n 246.58333, 246.75 , 246.91667, 247.08333, 247.25 , 247.41667,\n 247.58333, 247.75 , 247.91667, 248.08333, 248.25 , 248.41667,\n 248.58333, 248.75 , 248.91667, 249.08333, 249.25 , 249.41667,\n 249.58333, 249.75 , 249.91667, 250.08333, 250.25 , 250.41667,\n 250.58333, 250.75 , 250.91667, 251.08333, 251.25 , 251.41667,\n 251.58333, 251.75 , 251.91667, 252.08333, 252.25 , 252.41667,\n 252.58333, 252.75 , 252.91667, 253.08333, 253.25 , 253.41667,\n 253.58333, 253.75 , 253.91667, 254.08333, 254.25 , 254.41667,\n 254.58333, 254.75 , 254.91667, 255.08333, 255.25 , 255.41667,\n 255.58333, 255.75 , 255.91667, 256.08334, 256.25 , 256.41666,\n 256.58334, 256.75 , 256.91666, 257.08334, 257.25 , 257.41666,\n 257.58334, 257.75 , 257.91666, 258.08334, 258.25 , 258.41666,\n 258.58334, 258.75 , 258.91666, 259.08334, 259.25 , 259.41666,\n 259.58334, 259.75 , 259.91666, 260.08334, 260.25 , 260.41666],\n dtype=float32)Latitude(Latitude)float3215.92 16.08 16.25 ... 35.58 35.75standard_name :latitudeunits :degrees_northpoint_spacing :evenlong_name :latitudeaxis :Ybounds :Lat_boundsarray([15.916667, 16.083334, 16.25 , 16.416666, 16.583334, 16.75 ,\n 16.916666, 17.083334, 17.25 , 17.416666, 17.583334, 17.75 ,\n 17.916666, 18.083334, 18.25 , 18.416666, 18.583334, 18.75 ,\n 18.916666, 19.083334, 19.25 , 19.416666, 19.583334, 19.75 ,\n 19.916666, 20.083334, 20.25 , 20.416666, 20.583334, 20.75 ,\n 20.916666, 21.083334, 21.25 , 21.416666, 21.583334, 21.75 ,\n 21.916666, 22.083334, 22.25 , 22.416666, 22.583334, 22.75 ,\n 22.916666, 23.083334, 23.25 , 23.416666, 23.583334, 23.75 ,\n 23.916666, 24.083334, 24.25 , 24.416666, 24.583334, 24.75 ,\n 24.916666, 25.083334, 25.25 , 25.416666, 25.583334, 25.75 ,\n 25.916666, 26.083334, 26.25 , 26.416666, 26.583334, 26.75 ,\n 26.916666, 27.083334, 27.25 , 27.416666, 27.583334, 27.75 ,\n 27.916666, 28.083334, 28.25 , 28.416666, 28.583334, 28.75 ,\n 28.916666, 29.083334, 29.25 , 29.416666, 29.583334, 29.75 ,\n 29.916666, 30.083334, 30.25 , 30.416666, 30.583334, 30.75 ,\n 30.916666, 31.083334, 31.25 , 31.416666, 31.583334, 31.75 ,\n 31.916666, 32.083332, 32.25 , 32.416668, 32.583332, 32.75 ,\n 32.916668, 33.083332, 33.25 , 33.416668, 33.583332, 33.75 ,\n 33.916668, 34.083332, 34.25 , 34.416668, 34.583332, 34.75 ,\n 34.916668, 35.083332, 35.25 , 35.416668, 35.583332, 35.75 ],\n dtype=float32)Time(Time)datetime64[ns]2021-07-05T12:00:00 ... 2021-09-...standard_name :timelong_name :Timebounds :Time_boundsaxis :Tarray(['2021-07-05T12:00:00.000000000', '2021-07-10T12:00:00.000000000',\n '2021-07-15T12:00:00.000000000', '2021-07-20T12:00:00.000000000',\n '2021-07-25T12:00:00.000000000', '2021-07-30T12:00:00.000000000',\n '2021-08-04T12:00:00.000000000', '2021-08-09T12:00:00.000000000',\n '2021-08-14T12:00:00.000000000', '2021-08-19T12:00:00.000000000',\n '2021-08-24T12:00:00.000000000', '2021-08-29T12:00:00.000000000',\n '2021-09-03T12:00:00.000000000', '2021-09-08T12:00:00.000000000',\n '2021-09-13T12:00:00.000000000', '2021-09-18T12:00:00.000000000',\n '2021-09-23T12:00:00.000000000', '2021-09-28T12:00:00.000000000'],\n dtype='datetime64[ns]')Indexes: (3)LongitudePandasIndexPandasIndex(Index([234.5833282470703, 234.75, 234.9166717529297,\n 235.0833282470703, 235.25, 235.4166717529297,\n 235.5833282470703, 235.75, 235.9166717529297,\n 236.0833282470703,\n ...\n 258.9166564941406, 259.0833435058594, 259.25,\n 259.4166564941406, 259.5833435058594, 259.75,\n 259.9166564941406, 260.0833435058594, 260.25,\n 260.4166564941406],\n dtype='float32', name='Longitude', length=156))LatitudePandasIndexPandasIndex(Index([15.916666984558105, 16.08333396911621, 16.25,\n 16.41666603088379, 16.58333396911621, 16.75,\n 16.91666603088379, 17.08333396911621, 17.25,\n 17.41666603088379,\n ...\n 34.25, 34.41666793823242, 34.58333206176758,\n 34.75, 34.91666793823242, 35.08333206176758,\n 35.25, 35.41666793823242, 35.58333206176758,\n 35.75],\n dtype='float32', name='Latitude', length=120))TimePandasIndexPandasIndex(DatetimeIndex(['2021-07-05 12:00:00', '2021-07-10 12:00:00',\n '2021-07-15 12:00:00', '2021-07-20 12:00:00',\n '2021-07-25 12:00:00', '2021-07-30 12:00:00',\n '2021-08-04 12:00:00', '2021-08-09 12:00:00',\n '2021-08-14 12:00:00', '2021-08-19 12:00:00',\n '2021-08-24 12:00:00', '2021-08-29 12:00:00',\n '2021-09-03 12:00:00', '2021-09-08 12:00:00',\n '2021-09-13 12:00:00', '2021-09-18 12:00:00',\n '2021-09-23 12:00:00', '2021-09-28 12:00:00'],\n dtype='datetime64[ns]', name='Time', freq=None))Attributes: (4)units :mlong_name :Sea Level Anomaly Estimatestandard_name :sea_surface_height_above_sea_levelalias :sea_surface_height_above_sea_level\n\n\n\nds_from_scratch.hvplot.image(x='Longitude', y='Latitude', cmap='RdBu', clim=(-0.5, 0.5), title=\"Sea Level Anomaly Estimate (m)\")\n\n\n\n\n\n \n\n\n\n\n\n\n\nIf we decide this is a file we want to keep around for a longer time period, we can move it to our persistent bucket. We can even make a subdirectory in our persistent bucket to keep us organized:\n\npersistent_dest_dir = f\"{persistent}/my-analysis-data/\"\n\n# Make directory in persistent bucket\ns3.mkdir(persistent_dest_dir)\n\n# Move the file\ns3.mv(scratch_nc_file_path, persistent_dest_dir)\n\n# Check the scratch and persistent bucket listings:\ns3.ls(scratch)\n\n['openscapeshub-scratch/ateucher/.placeholder']\n\n\n\ns3.ls(persistent)\n\n['openscapeshub-persistent/ateucher/.placeholder',\n 'openscapeshub-persistent/ateucher/my-analysis-data']\n\n\n\ns3.ls(persistent_dest_dir)\n\n['openscapeshub-persistent/ateucher/my-analysis-data/test123.nc']\n\n\n\n\n\nYou may already have some data in your HOME directory that you would like to move out to a persistent bucket. You can do that using the awsv2 s3 command line tool, which is already installed on the hub. You can open a terminal from the Hub Launcher - it will open in your HOME directory. You can then use the awsv2 s3 mv command to move a file to your bucket.\n\n\n$ awsv2 s3 mv my-big-file.nc $PERSISTENT_BUCKET/ # The trailing slash is important here\nmove: ./my-big-file.nc to s3://openscapeshub-persistent/ateucher/my-big-file.nc\n\n\n\nList the contents of the local results-data directory:\n$ ls results-data/\nmy-big-file1.nc my-big-file2.nc\nUse awsv2 s3 mv with the --recursive flag to move all files in a directory to a new directory in PERSISTENT_BUCKET\n$ awsv2 s3 mv --recursive results-data $PERSISTENT_BUCKET/results-data/\nmove: results-data/my-big-file1.nc to s3://openscapeshub-persistent/ateucher/results-data/my-big-file1.nc\nmove: results-data/my-big-file2.nc to s3://openscapeshub-persistent/ateucher/results-data/my-big-file2.nc" + "text": "The Global Ecosystem Dynamics Investigation (GEDI) Level 4A Footprint Level Aboveground Biomass Density (AGBD) dataset provides predictions of the aboveground biomass density (AGBD; in Mg/ha) and estimates of the prediction standard error within each sampled geolocated GEDI footprint. GEDI L4A dataset is available for the period starting 2019-04-17 and covers 52 N to 52 S latitudes. GEDI L4A data files are natively in HDF5 format.\nThis tutorial will demonstrate how to compute the distribution of carbon stocks or AGBD for a protected area using the GEDI L4A dataset." }, { - "objectID": "how-tos/using-s3-storage.html#reading-and-writing-to-the-scratch_bucket", - "href": "how-tos/using-s3-storage.html#reading-and-writing-to-the-scratch_bucket", - "title": "Using S3 Bucket Storage in NASA-Openscapes Hub", + "objectID": "tutorials/Above_Ground_Biomass_Density.html#overview", + "href": "tutorials/Above_Ground_Biomass_Density.html#overview", + "title": "Distribution of carbon stocks in a protected area", "section": "", - "text": "We will start by accessing the same data we did in the Earthdata Cloud Clinic - reading it into memory as an xarray object and subsetting it.\n\nimport earthaccess \nimport xarray as xr\nimport hvplot.xarray #plot\nimport os\nimport tempfile\nimport s3fs # aws s3 access\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\nauth = earthaccess.login()\n\n\ndata_name = \"SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205\"\n\nresults = earthaccess.search_data(\n short_name=data_name,\n cloud_hosted=True,\n temporal=(\"2021-07-01\", \"2021-09-30\"),\n)\n\nGranules found: 18\n\n\n\nds = xr.open_mfdataset(earthaccess.open(results))\nds\n\nOpening 18 granules, approx size: 0.16 GB\nusing endpoint: https://archive.podaac.earthdata.nasa.gov/s3credentials\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset> Size: 299MB\nDimensions: (Time: 18, Longitude: 2160, nv: 2, Latitude: 960)\nCoordinates:\n * Longitude (Longitude) float32 9kB 0.08333 0.25 0.4167 ... 359.8 359.9\n * Latitude (Latitude) float32 4kB -79.92 -79.75 -79.58 ... 79.75 79.92\n * Time (Time) datetime64[ns] 144B 2021-07-05T12:00:00 ... 2021-09-2...\nDimensions without coordinates: nv\nData variables:\n Lon_bounds (Time, Longitude, nv) float32 311kB dask.array<chunksize=(1, 2160, 2), meta=np.ndarray>\n Lat_bounds (Time, Latitude, nv) float32 138kB dask.array<chunksize=(1, 960, 2), meta=np.ndarray>\n Time_bounds (Time, nv) datetime64[ns] 288B dask.array<chunksize=(1, 2), meta=np.ndarray>\n SLA (Time, Latitude, Longitude) float32 149MB dask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n SLA_ERR (Time, Latitude, Longitude) float32 149MB dask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\nAttributes: (12/21)\n Conventions: CF-1.6\n ncei_template_version: NCEI_NetCDF_Grid_Template_v2.0\n Institution: Jet Propulsion Laboratory\n geospatial_lat_min: -79.916664\n geospatial_lat_max: 79.916664\n geospatial_lon_min: 0.083333336\n ... ...\n version_number: 2205\n Data_Pnts_Each_Sat: {\"16\": 743215, \"1007\": 674076}\n source_version: commit 58c7da13c0c0069ae940c33a82bf1544b7d991bf\n SLA_Global_MEAN: 0.06428374482174487\n SLA_Global_STD: 0.0905195660534004\n latency: finalxarray.DatasetDimensions:Time: 18Longitude: 2160nv: 2Latitude: 960Coordinates: (3)Longitude(Longitude)float320.08333 0.25 0.4167 ... 359.8 359.9standard_name :longitudeunits :degrees_eastpoint_spacing :evenlong_name :longitudeaxis :Xbounds :Lon_boundsarray([8.333334e-02, 2.500000e-01, 4.166667e-01, ..., 3.595833e+02,\n 3.597500e+02, 3.599167e+02], dtype=float32)Latitude(Latitude)float32-79.92 -79.75 ... 79.75 79.92standard_name :latitudeunits :degrees_northpoint_spacing :evenlong_name :latitudeaxis :Ybounds :Lat_boundsarray([-79.916664, -79.75 , -79.583336, ..., 79.583336, 79.75 ,\n 79.916664], dtype=float32)Time(Time)datetime64[ns]2021-07-05T12:00:00 ... 2021-09-...standard_name :timelong_name :Timebounds :Time_boundsaxis :Tarray(['2021-07-05T12:00:00.000000000', '2021-07-10T12:00:00.000000000',\n '2021-07-15T12:00:00.000000000', '2021-07-20T12:00:00.000000000',\n '2021-07-25T12:00:00.000000000', '2021-07-30T12:00:00.000000000',\n '2021-08-04T12:00:00.000000000', '2021-08-09T12:00:00.000000000',\n '2021-08-14T12:00:00.000000000', '2021-08-19T12:00:00.000000000',\n '2021-08-24T12:00:00.000000000', '2021-08-29T12:00:00.000000000',\n '2021-09-03T12:00:00.000000000', '2021-09-08T12:00:00.000000000',\n '2021-09-13T12:00:00.000000000', '2021-09-18T12:00:00.000000000',\n '2021-09-23T12:00:00.000000000', '2021-09-28T12:00:00.000000000'],\n dtype='datetime64[ns]')Data variables: (5)Lon_bounds(Time, Longitude, nv)float32dask.array<chunksize=(1, 2160, 2), meta=np.ndarray>units :degrees_eastcomment :longitude values at the west and east bounds of each pixel.\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n303.75 kiB\n16.88 kiB\n\n\nShape\n(18, 2160, 2)\n(1, 2160, 2)\n\n\nDask graph\n18 chunks in 55 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\nLat_bounds\n\n\n(Time, Latitude, nv)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2), meta=np.ndarray>\n\n\n\n\nunits :\n\ndegrees_north\n\ncomment :\n\nlatitude values at the north and south bounds of each pixel.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n135.00 kiB\n7.50 kiB\n\n\nShape\n(18, 960, 2)\n(1, 960, 2)\n\n\nDask graph\n18 chunks in 55 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nTime_bounds\n\n\n(Time, nv)\n\n\ndatetime64[ns]\n\n\ndask.array<chunksize=(1, 2), meta=np.ndarray>\n\n\n\n\ncomment :\n\nTime bounds for each time value, same value as time variable. The time variable is defined on points instead of on bounding boxes.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n288 B\n16 B\n\n\nShape\n(18, 2)\n(1, 2)\n\n\nDask graph\n18 chunks in 37 graph layers\n\n\nData type\ndatetime64[ns] numpy.ndarray\n\n\n\n\n\n\n\n\n\nSLA\n\n\n(Time, Latitude, Longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n\n\n\n\nunits :\n\nm\n\nlong_name :\n\nSea Level Anomaly Estimate\n\nstandard_name :\n\nsea_surface_height_above_sea_level\n\nalias :\n\nsea_surface_height_above_sea_level\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n142.38 MiB\n7.91 MiB\n\n\nShape\n(18, 960, 2160)\n(1, 960, 2160)\n\n\nDask graph\n18 chunks in 37 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nSLA_ERR\n\n\n(Time, Latitude, Longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n\n\n\n\nunits :\n\nm\n\nlong_name :\n\nSea Level Anomaly Error Estimate\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n142.38 MiB\n7.91 MiB\n\n\nShape\n(18, 960, 2160)\n(1, 960, 2160)\n\n\nDask graph\n18 chunks in 37 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\nIndexes: (3)LongitudePandasIndexPandasIndex(Index([0.0833333358168602, 0.25, 0.4166666567325592,\n 0.5833333134651184, 0.75, 0.9166666865348816,\n 1.0833333730697632, 1.25, 1.4166666269302368,\n 1.5833333730697632,\n ...\n 358.4166564941406, 358.5833435058594, 358.75,\n 358.9166564941406, 359.0833435058594, 359.25,\n 359.4166564941406, 359.5833435058594, 359.75,\n 359.9166564941406],\n dtype='float32', name='Longitude', length=2160))LatitudePandasIndexPandasIndex(Index([-79.91666412353516, -79.75, -79.58333587646484,\n -79.41666412353516, -79.25, -79.08333587646484,\n -78.91666412353516, -78.75, -78.58333587646484,\n -78.41666412353516,\n ...\n 78.41666412353516, 78.58333587646484, 78.75,\n 78.91666412353516, 79.08333587646484, 79.25,\n 79.41666412353516, 79.58333587646484, 79.75,\n 79.91666412353516],\n dtype='float32', name='Latitude', length=960))TimePandasIndexPandasIndex(DatetimeIndex(['2021-07-05 12:00:00', '2021-07-10 12:00:00',\n '2021-07-15 12:00:00', '2021-07-20 12:00:00',\n '2021-07-25 12:00:00', '2021-07-30 12:00:00',\n '2021-08-04 12:00:00', '2021-08-09 12:00:00',\n '2021-08-14 12:00:00', '2021-08-19 12:00:00',\n '2021-08-24 12:00:00', '2021-08-29 12:00:00',\n '2021-09-03 12:00:00', '2021-09-08 12:00:00',\n '2021-09-13 12:00:00', '2021-09-18 12:00:00',\n '2021-09-23 12:00:00', '2021-09-28 12:00:00'],\n dtype='datetime64[ns]', name='Time', freq=None))Attributes: (21)Conventions :CF-1.6ncei_template_version :NCEI_NetCDF_Grid_Template_v2.0Institution :Jet Propulsion Laboratorygeospatial_lat_min :-79.916664geospatial_lat_max :79.916664geospatial_lon_min :0.083333336geospatial_lon_max :359.91666time_coverage_start :2021-07-05time_coverage_end :2021-07-05date_created :2022-10-30T20:58:16.297288title :Sea Level Anomaly Estimate based on Altimeter Data, final product (replaced interim version).short_name :SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205long_name :MEaSUREs Gridded Sea Surface Height Anomalies Version 2205summary :Sea level anomaly grids from altimeter data using Kriging interpolation, which gives best linear prediction based upon prior knowledge of covariance. DOI :10.5067/SLREF-CDRV3version_number :2205Data_Pnts_Each_Sat :{\"16\": 743215, \"1007\": 674076}source_version :commit 58c7da13c0c0069ae940c33a82bf1544b7d991bfSLA_Global_MEAN :0.06428374482174487SLA_Global_STD :0.0905195660534004latency :final\n\n\n\nds_subset = ds['SLA'].sel(Latitude=slice(15.8, 35.9), Longitude=slice(234.5,260.5)) \nds_subset\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray 'SLA' (Time: 18, Latitude: 120, Longitude: 156)> Size: 1MB\ndask.array<getitem, shape=(18, 120, 156), dtype=float32, chunksize=(1, 120, 156), chunktype=numpy.ndarray>\nCoordinates:\n * Longitude (Longitude) float32 624B 234.6 234.8 234.9 ... 260.1 260.2 260.4\n * Latitude (Latitude) float32 480B 15.92 16.08 16.25 ... 35.42 35.58 35.75\n * Time (Time) datetime64[ns] 144B 2021-07-05T12:00:00 ... 2021-09-28T...\nAttributes:\n units: m\n long_name: Sea Level Anomaly Estimate\n standard_name: sea_surface_height_above_sea_level\n alias: sea_surface_height_above_sea_levelxarray.DataArray'SLA'Time: 18Latitude: 120Longitude: 156dask.array<chunksize=(1, 120, 156), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n1.29 MiB\n73.12 kiB\n\n\nShape\n(18, 120, 156)\n(1, 120, 156)\n\n\nDask graph\n18 chunks in 38 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\nCoordinates: (3)Longitude(Longitude)float32234.6 234.8 234.9 ... 260.2 260.4standard_name :longitudeunits :degrees_eastpoint_spacing :evenlong_name :longitudeaxis :Xbounds :Lon_boundsarray([234.58333, 234.75 , 234.91667, 235.08333, 235.25 , 235.41667,\n 235.58333, 235.75 , 235.91667, 236.08333, 236.25 , 236.41667,\n 236.58333, 236.75 , 236.91667, 237.08333, 237.25 , 237.41667,\n 237.58333, 237.75 , 237.91667, 238.08333, 238.25 , 238.41667,\n 238.58333, 238.75 , 238.91667, 239.08333, 239.25 , 239.41667,\n 239.58333, 239.75 , 239.91667, 240.08333, 240.25 , 240.41667,\n 240.58333, 240.75 , 240.91667, 241.08333, 241.25 , 241.41667,\n 241.58333, 241.75 , 241.91667, 242.08333, 242.25 , 242.41667,\n 242.58333, 242.75 , 242.91667, 243.08333, 243.25 , 243.41667,\n 243.58333, 243.75 , 243.91667, 244.08333, 244.25 , 244.41667,\n 244.58333, 244.75 , 244.91667, 245.08333, 245.25 , 245.41667,\n 245.58333, 245.75 , 245.91667, 246.08333, 246.25 , 246.41667,\n 246.58333, 246.75 , 246.91667, 247.08333, 247.25 , 247.41667,\n 247.58333, 247.75 , 247.91667, 248.08333, 248.25 , 248.41667,\n 248.58333, 248.75 , 248.91667, 249.08333, 249.25 , 249.41667,\n 249.58333, 249.75 , 249.91667, 250.08333, 250.25 , 250.41667,\n 250.58333, 250.75 , 250.91667, 251.08333, 251.25 , 251.41667,\n 251.58333, 251.75 , 251.91667, 252.08333, 252.25 , 252.41667,\n 252.58333, 252.75 , 252.91667, 253.08333, 253.25 , 253.41667,\n 253.58333, 253.75 , 253.91667, 254.08333, 254.25 , 254.41667,\n 254.58333, 254.75 , 254.91667, 255.08333, 255.25 , 255.41667,\n 255.58333, 255.75 , 255.91667, 256.08334, 256.25 , 256.41666,\n 256.58334, 256.75 , 256.91666, 257.08334, 257.25 , 257.41666,\n 257.58334, 257.75 , 257.91666, 258.08334, 258.25 , 258.41666,\n 258.58334, 258.75 , 258.91666, 259.08334, 259.25 , 259.41666,\n 259.58334, 259.75 , 259.91666, 260.08334, 260.25 , 260.41666],\n dtype=float32)Latitude(Latitude)float3215.92 16.08 16.25 ... 35.58 35.75standard_name :latitudeunits :degrees_northpoint_spacing :evenlong_name :latitudeaxis :Ybounds :Lat_boundsarray([15.916667, 16.083334, 16.25 , 16.416666, 16.583334, 16.75 ,\n 16.916666, 17.083334, 17.25 , 17.416666, 17.583334, 17.75 ,\n 17.916666, 18.083334, 18.25 , 18.416666, 18.583334, 18.75 ,\n 18.916666, 19.083334, 19.25 , 19.416666, 19.583334, 19.75 ,\n 19.916666, 20.083334, 20.25 , 20.416666, 20.583334, 20.75 ,\n 20.916666, 21.083334, 21.25 , 21.416666, 21.583334, 21.75 ,\n 21.916666, 22.083334, 22.25 , 22.416666, 22.583334, 22.75 ,\n 22.916666, 23.083334, 23.25 , 23.416666, 23.583334, 23.75 ,\n 23.916666, 24.083334, 24.25 , 24.416666, 24.583334, 24.75 ,\n 24.916666, 25.083334, 25.25 , 25.416666, 25.583334, 25.75 ,\n 25.916666, 26.083334, 26.25 , 26.416666, 26.583334, 26.75 ,\n 26.916666, 27.083334, 27.25 , 27.416666, 27.583334, 27.75 ,\n 27.916666, 28.083334, 28.25 , 28.416666, 28.583334, 28.75 ,\n 28.916666, 29.083334, 29.25 , 29.416666, 29.583334, 29.75 ,\n 29.916666, 30.083334, 30.25 , 30.416666, 30.583334, 30.75 ,\n 30.916666, 31.083334, 31.25 , 31.416666, 31.583334, 31.75 ,\n 31.916666, 32.083332, 32.25 , 32.416668, 32.583332, 32.75 ,\n 32.916668, 33.083332, 33.25 , 33.416668, 33.583332, 33.75 ,\n 33.916668, 34.083332, 34.25 , 34.416668, 34.583332, 34.75 ,\n 34.916668, 35.083332, 35.25 , 35.416668, 35.583332, 35.75 ],\n dtype=float32)Time(Time)datetime64[ns]2021-07-05T12:00:00 ... 2021-09-...standard_name :timelong_name :Timebounds :Time_boundsaxis :Tarray(['2021-07-05T12:00:00.000000000', '2021-07-10T12:00:00.000000000',\n '2021-07-15T12:00:00.000000000', '2021-07-20T12:00:00.000000000',\n '2021-07-25T12:00:00.000000000', '2021-07-30T12:00:00.000000000',\n '2021-08-04T12:00:00.000000000', '2021-08-09T12:00:00.000000000',\n '2021-08-14T12:00:00.000000000', '2021-08-19T12:00:00.000000000',\n '2021-08-24T12:00:00.000000000', '2021-08-29T12:00:00.000000000',\n '2021-09-03T12:00:00.000000000', '2021-09-08T12:00:00.000000000',\n '2021-09-13T12:00:00.000000000', '2021-09-18T12:00:00.000000000',\n '2021-09-23T12:00:00.000000000', '2021-09-28T12:00:00.000000000'],\n dtype='datetime64[ns]')Indexes: (3)LongitudePandasIndexPandasIndex(Index([234.5833282470703, 234.75, 234.9166717529297,\n 235.0833282470703, 235.25, 235.4166717529297,\n 235.5833282470703, 235.75, 235.9166717529297,\n 236.0833282470703,\n ...\n 258.9166564941406, 259.0833435058594, 259.25,\n 259.4166564941406, 259.5833435058594, 259.75,\n 259.9166564941406, 260.0833435058594, 260.25,\n 260.4166564941406],\n dtype='float32', name='Longitude', length=156))LatitudePandasIndexPandasIndex(Index([15.916666984558105, 16.08333396911621, 16.25,\n 16.41666603088379, 16.58333396911621, 16.75,\n 16.91666603088379, 17.08333396911621, 17.25,\n 17.41666603088379,\n ...\n 34.25, 34.41666793823242, 34.58333206176758,\n 34.75, 34.91666793823242, 35.08333206176758,\n 35.25, 35.41666793823242, 35.58333206176758,\n 35.75],\n dtype='float32', name='Latitude', length=120))TimePandasIndexPandasIndex(DatetimeIndex(['2021-07-05 12:00:00', '2021-07-10 12:00:00',\n '2021-07-15 12:00:00', '2021-07-20 12:00:00',\n '2021-07-25 12:00:00', '2021-07-30 12:00:00',\n '2021-08-04 12:00:00', '2021-08-09 12:00:00',\n '2021-08-14 12:00:00', '2021-08-19 12:00:00',\n '2021-08-24 12:00:00', '2021-08-29 12:00:00',\n '2021-09-03 12:00:00', '2021-09-08 12:00:00',\n '2021-09-13 12:00:00', '2021-09-18 12:00:00',\n '2021-09-23 12:00:00', '2021-09-28 12:00:00'],\n dtype='datetime64[ns]', name='Time', freq=None))Attributes: (4)units :mlong_name :Sea Level Anomaly Estimatestandard_name :sea_surface_height_above_sea_levelalias :sea_surface_height_above_sea_level" + "text": "The Global Ecosystem Dynamics Investigation (GEDI) Level 4A Footprint Level Aboveground Biomass Density (AGBD) dataset provides predictions of the aboveground biomass density (AGBD; in Mg/ha) and estimates of the prediction standard error within each sampled geolocated GEDI footprint. GEDI L4A dataset is available for the period starting 2019-04-17 and covers 52 N to 52 S latitudes. GEDI L4A data files are natively in HDF5 format.\nThis tutorial will demonstrate how to compute the distribution of carbon stocks or AGBD for a protected area using the GEDI L4A dataset." }, { - "objectID": "how-tos/using-s3-storage.html#home-directory", - "href": "how-tos/using-s3-storage.html#home-directory", - "title": "Using S3 Bucket Storage in NASA-Openscapes Hub", - "section": "", - "text": "Imagining this ds_subset object is now an important intermediate dataset, or the result of a complex analysis and we want to save it. Our default action might be to just save it to our HOME directory. This is simple, but we want to avoid this as it incurs significant storage costs, and using this data later will be slow.\nds_subset.to_netcdf(\"test.nc\") # avoid writing to home directory like this" + "objectID": "tutorials/Above_Ground_Biomass_Density.html#learning-objectives", + "href": "tutorials/Above_Ground_Biomass_Density.html#learning-objectives", + "title": "Distribution of carbon stocks in a protected area", + "section": "Learning Objectives", + "text": "Learning Objectives\n\nUse the earthaccess Python library to search for GEDI L4A granules for time and area of interest. The earthaccess library simplifies access to NASA’s Common Metadata Repository (CMR) Application Programming Interface (API).\nUse NASA OPeNDAP Hyrax to retrieve the GEDI L4A dataset. Hyrax allows direct access to selected variables for the dataset within the spatial-temporal bounds without having to download the whole data file.\nCompute distributions of AGBD across various plant functional types (PFTs) in a study area." }, { - "objectID": "how-tos/using-s3-storage.html#use-the-s3fs-package-to-interact-with-our-s3-bucket.", - "href": "how-tos/using-s3-storage.html#use-the-s3fs-package-to-interact-with-our-s3-bucket.", - "title": "Using S3 Bucket Storage in NASA-Openscapes Hub", - "section": "", - "text": "s3fs is a Python library that allows us to interact with S3 objects in a file-system like manner.\n\n# Create a S3FileSystem class\ns3 = s3fs.S3FileSystem()\n\n# Get scratch and persistent buckets\nscratch = os.environ[\"SCRATCH_BUCKET\"]\npersistent = os.environ[\"PERSISTENT_BUCKET\"]\n\nprint(scratch)\nprint(persistent)\n\ns3://openscapeshub-scratch/ateucher\ns3://openscapeshub-persistent/ateucher\n\n\nOur user-specific directories in the two buckets aren’t actually created until we put something in them, so if we try to check for their existence or list their contents before they are created, we will get an error. We will use the S3FileSystem.touch() method to place a simple empty file called .placeholder in each one to bring them into existence.\n\ns3.touch(f\"{scratch}/.placeholder\")\n\ns3.ls(scratch)\n\n['openscapeshub-scratch/ateucher/.placeholder']\n\n\nand in our persistent bucket:\n\ns3.touch(f\"{persistent}/.placeholder\")\n\ns3.ls(persistent)\n\n['openscapeshub-persistent/ateucher/.placeholder']\n\n\n(Note that adding these placeholders isn’t strictly necessary, as the first time you write anything to these buckets they will be created.)" + "objectID": "tutorials/Above_Ground_Biomass_Density.html#requirements", + "href": "tutorials/Above_Ground_Biomass_Density.html#requirements", + "title": "Distribution of carbon stocks in a protected area", + "section": "Requirements", + "text": "Requirements\n\n1. Compute environment\nThis notebook can be run in any personal computing environment (e.g., desktop/laptops), on-premise solution (e.g., High-Performance Computing), or on the Cloud (e.g., Amazon Web Service). ### 2. Earthdata Login An Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up. ### 3. Additional Requirements\nThis tutorial requires the following Python modules installed in your system: earthaccess, requests, netCDF4, datetime, dask, pandas, geopandas, contextily, shapely, seaborn. To install the necessary Python modules, you can run:\npip install -r earthaccess requests netCDF4 datetime dask pandas geopandas contextily shapely seaborn" }, { - "objectID": "how-tos/using-s3-storage.html#save-dataset-as-netcdf-file-in-scratch-bucket", - "href": "how-tos/using-s3-storage.html#save-dataset-as-netcdf-file-in-scratch-bucket", - "title": "Using S3 Bucket Storage in NASA-Openscapes Hub", - "section": "", - "text": "Next we can save ds_subset as a netcdf file in our scratch bucket. This involves writing to a temporary directory first, and then moving that to the SCRATCH bucket:\n\n# Where we want to store it:\nscratch_nc_file_path = f\"{scratch}/test123.nc\"\n\n# Create a temporary intermediate file and save it to the bucket\nwith tempfile.NamedTemporaryFile(suffix = \".nc\") as tmp:\n ds_subset.to_netcdf(tmp.name) # save it to a temporary file\n s3.put(tmp.name, scratch_nc_file_path) # move that file to the scratch bucket\n\n# Ensure the file is there\ns3.ls(scratch)\n\n['openscapeshub-scratch/ateucher/.placeholder',\n 'openscapeshub-scratch/ateucher/test123.nc']\n\n\nAnd we can open it to ensure it worked:\n\nds_from_scratch = xr.open_dataarray(s3.open(scratch_nc_file_path))\n\nds_from_scratch\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray 'SLA' (Time: 18, Latitude: 120, Longitude: 156)> Size: 1MB\n[336960 values with dtype=float32]\nCoordinates:\n * Longitude (Longitude) float32 624B 234.6 234.8 234.9 ... 260.1 260.2 260.4\n * Latitude (Latitude) float32 480B 15.92 16.08 16.25 ... 35.42 35.58 35.75\n * Time (Time) datetime64[ns] 144B 2021-07-05T12:00:00 ... 2021-09-28T...\nAttributes:\n units: m\n long_name: Sea Level Anomaly Estimate\n standard_name: sea_surface_height_above_sea_level\n alias: sea_surface_height_above_sea_levelxarray.DataArray'SLA'Time: 18Latitude: 120Longitude: 156...[336960 values with dtype=float32]Coordinates: (3)Longitude(Longitude)float32234.6 234.8 234.9 ... 260.2 260.4standard_name :longitudeunits :degrees_eastpoint_spacing :evenlong_name :longitudeaxis :Xbounds :Lon_boundsarray([234.58333, 234.75 , 234.91667, 235.08333, 235.25 , 235.41667,\n 235.58333, 235.75 , 235.91667, 236.08333, 236.25 , 236.41667,\n 236.58333, 236.75 , 236.91667, 237.08333, 237.25 , 237.41667,\n 237.58333, 237.75 , 237.91667, 238.08333, 238.25 , 238.41667,\n 238.58333, 238.75 , 238.91667, 239.08333, 239.25 , 239.41667,\n 239.58333, 239.75 , 239.91667, 240.08333, 240.25 , 240.41667,\n 240.58333, 240.75 , 240.91667, 241.08333, 241.25 , 241.41667,\n 241.58333, 241.75 , 241.91667, 242.08333, 242.25 , 242.41667,\n 242.58333, 242.75 , 242.91667, 243.08333, 243.25 , 243.41667,\n 243.58333, 243.75 , 243.91667, 244.08333, 244.25 , 244.41667,\n 244.58333, 244.75 , 244.91667, 245.08333, 245.25 , 245.41667,\n 245.58333, 245.75 , 245.91667, 246.08333, 246.25 , 246.41667,\n 246.58333, 246.75 , 246.91667, 247.08333, 247.25 , 247.41667,\n 247.58333, 247.75 , 247.91667, 248.08333, 248.25 , 248.41667,\n 248.58333, 248.75 , 248.91667, 249.08333, 249.25 , 249.41667,\n 249.58333, 249.75 , 249.91667, 250.08333, 250.25 , 250.41667,\n 250.58333, 250.75 , 250.91667, 251.08333, 251.25 , 251.41667,\n 251.58333, 251.75 , 251.91667, 252.08333, 252.25 , 252.41667,\n 252.58333, 252.75 , 252.91667, 253.08333, 253.25 , 253.41667,\n 253.58333, 253.75 , 253.91667, 254.08333, 254.25 , 254.41667,\n 254.58333, 254.75 , 254.91667, 255.08333, 255.25 , 255.41667,\n 255.58333, 255.75 , 255.91667, 256.08334, 256.25 , 256.41666,\n 256.58334, 256.75 , 256.91666, 257.08334, 257.25 , 257.41666,\n 257.58334, 257.75 , 257.91666, 258.08334, 258.25 , 258.41666,\n 258.58334, 258.75 , 258.91666, 259.08334, 259.25 , 259.41666,\n 259.58334, 259.75 , 259.91666, 260.08334, 260.25 , 260.41666],\n dtype=float32)Latitude(Latitude)float3215.92 16.08 16.25 ... 35.58 35.75standard_name :latitudeunits :degrees_northpoint_spacing :evenlong_name :latitudeaxis :Ybounds :Lat_boundsarray([15.916667, 16.083334, 16.25 , 16.416666, 16.583334, 16.75 ,\n 16.916666, 17.083334, 17.25 , 17.416666, 17.583334, 17.75 ,\n 17.916666, 18.083334, 18.25 , 18.416666, 18.583334, 18.75 ,\n 18.916666, 19.083334, 19.25 , 19.416666, 19.583334, 19.75 ,\n 19.916666, 20.083334, 20.25 , 20.416666, 20.583334, 20.75 ,\n 20.916666, 21.083334, 21.25 , 21.416666, 21.583334, 21.75 ,\n 21.916666, 22.083334, 22.25 , 22.416666, 22.583334, 22.75 ,\n 22.916666, 23.083334, 23.25 , 23.416666, 23.583334, 23.75 ,\n 23.916666, 24.083334, 24.25 , 24.416666, 24.583334, 24.75 ,\n 24.916666, 25.083334, 25.25 , 25.416666, 25.583334, 25.75 ,\n 25.916666, 26.083334, 26.25 , 26.416666, 26.583334, 26.75 ,\n 26.916666, 27.083334, 27.25 , 27.416666, 27.583334, 27.75 ,\n 27.916666, 28.083334, 28.25 , 28.416666, 28.583334, 28.75 ,\n 28.916666, 29.083334, 29.25 , 29.416666, 29.583334, 29.75 ,\n 29.916666, 30.083334, 30.25 , 30.416666, 30.583334, 30.75 ,\n 30.916666, 31.083334, 31.25 , 31.416666, 31.583334, 31.75 ,\n 31.916666, 32.083332, 32.25 , 32.416668, 32.583332, 32.75 ,\n 32.916668, 33.083332, 33.25 , 33.416668, 33.583332, 33.75 ,\n 33.916668, 34.083332, 34.25 , 34.416668, 34.583332, 34.75 ,\n 34.916668, 35.083332, 35.25 , 35.416668, 35.583332, 35.75 ],\n dtype=float32)Time(Time)datetime64[ns]2021-07-05T12:00:00 ... 2021-09-...standard_name :timelong_name :Timebounds :Time_boundsaxis :Tarray(['2021-07-05T12:00:00.000000000', '2021-07-10T12:00:00.000000000',\n '2021-07-15T12:00:00.000000000', '2021-07-20T12:00:00.000000000',\n '2021-07-25T12:00:00.000000000', '2021-07-30T12:00:00.000000000',\n '2021-08-04T12:00:00.000000000', '2021-08-09T12:00:00.000000000',\n '2021-08-14T12:00:00.000000000', '2021-08-19T12:00:00.000000000',\n '2021-08-24T12:00:00.000000000', '2021-08-29T12:00:00.000000000',\n '2021-09-03T12:00:00.000000000', '2021-09-08T12:00:00.000000000',\n '2021-09-13T12:00:00.000000000', '2021-09-18T12:00:00.000000000',\n '2021-09-23T12:00:00.000000000', '2021-09-28T12:00:00.000000000'],\n dtype='datetime64[ns]')Indexes: (3)LongitudePandasIndexPandasIndex(Index([234.5833282470703, 234.75, 234.9166717529297,\n 235.0833282470703, 235.25, 235.4166717529297,\n 235.5833282470703, 235.75, 235.9166717529297,\n 236.0833282470703,\n ...\n 258.9166564941406, 259.0833435058594, 259.25,\n 259.4166564941406, 259.5833435058594, 259.75,\n 259.9166564941406, 260.0833435058594, 260.25,\n 260.4166564941406],\n dtype='float32', name='Longitude', length=156))LatitudePandasIndexPandasIndex(Index([15.916666984558105, 16.08333396911621, 16.25,\n 16.41666603088379, 16.58333396911621, 16.75,\n 16.91666603088379, 17.08333396911621, 17.25,\n 17.41666603088379,\n ...\n 34.25, 34.41666793823242, 34.58333206176758,\n 34.75, 34.91666793823242, 35.08333206176758,\n 35.25, 35.41666793823242, 35.58333206176758,\n 35.75],\n dtype='float32', name='Latitude', length=120))TimePandasIndexPandasIndex(DatetimeIndex(['2021-07-05 12:00:00', '2021-07-10 12:00:00',\n '2021-07-15 12:00:00', '2021-07-20 12:00:00',\n '2021-07-25 12:00:00', '2021-07-30 12:00:00',\n '2021-08-04 12:00:00', '2021-08-09 12:00:00',\n '2021-08-14 12:00:00', '2021-08-19 12:00:00',\n '2021-08-24 12:00:00', '2021-08-29 12:00:00',\n '2021-09-03 12:00:00', '2021-09-08 12:00:00',\n '2021-09-13 12:00:00', '2021-09-18 12:00:00',\n '2021-09-23 12:00:00', '2021-09-28 12:00:00'],\n dtype='datetime64[ns]', name='Time', freq=None))Attributes: (4)units :mlong_name :Sea Level Anomaly Estimatestandard_name :sea_surface_height_above_sea_levelalias :sea_surface_height_above_sea_level\n\n\n\nds_from_scratch.hvplot.image(x='Longitude', y='Latitude', cmap='RdBu', clim=(-0.5, 0.5), title=\"Sea Level Anomaly Estimate (m)\")" + "objectID": "tutorials/Above_Ground_Biomass_Density.html#import-packages", + "href": "tutorials/Above_Ground_Biomass_Density.html#import-packages", + "title": "Distribution of carbon stocks in a protected area", + "section": "Import packages", + "text": "Import packages\n\nimport earthaccess\nimport pandas as pd\nimport geopandas as gpd\nimport dask.dataframe as dd\nimport contextily as ctx\nimport netCDF4 as nc\nfrom os import path\nfrom matplotlib import pyplot as plt\nfrom shapely.ops import orient\nfrom IPython import display\nfrom requests.adapters import HTTPAdapter, Retry\nimport seaborn as sns\nsns.set(style='whitegrid')\n\nAccess to NASA OPeNDAP requires NASA Earthdata authentication. We recommend authenticating your Earthdata Login (EDL) information using the earthaccess python library as follows:\n\nauth = earthaccess.login(strategy=\"netrc\") # works if the EDL login already been persisted to a netrc\nif not auth.authenticated:\n # ask for EDL credentials and persist them in a .netrc file\n auth = earthaccess.login(strategy=\"interactive\", persist=True)\n\nYou're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 10/28/2023\nUsing .netrc file for EDL" }, { - "objectID": "how-tos/using-s3-storage.html#move-data-to-the-persistent-bucket", - "href": "how-tos/using-s3-storage.html#move-data-to-the-persistent-bucket", - "title": "Using S3 Bucket Storage in NASA-Openscapes Hub", - "section": "", - "text": "If we decide this is a file we want to keep around for a longer time period, we can move it to our persistent bucket. We can even make a subdirectory in our persistent bucket to keep us organized:\n\npersistent_dest_dir = f\"{persistent}/my-analysis-data/\"\n\n# Make directory in persistent bucket\ns3.mkdir(persistent_dest_dir)\n\n# Move the file\ns3.mv(scratch_nc_file_path, persistent_dest_dir)\n\n# Check the scratch and persistent bucket listings:\ns3.ls(scratch)\n\n['openscapeshub-scratch/ateucher/.placeholder']\n\n\n\ns3.ls(persistent)\n\n['openscapeshub-persistent/ateucher/.placeholder',\n 'openscapeshub-persistent/ateucher/my-analysis-data']\n\n\n\ns3.ls(persistent_dest_dir)\n\n['openscapeshub-persistent/ateucher/my-analysis-data/test123.nc']" + "objectID": "tutorials/Above_Ground_Biomass_Density.html#search-gedi-l4a-granules", + "href": "tutorials/Above_Ground_Biomass_Density.html#search-gedi-l4a-granules", + "title": "Distribution of carbon stocks in a protected area", + "section": "Search GEDI L4A granules", + "text": "Search GEDI L4A granules\nFirst, we will search for granules that overlay our area of interest. We will then access the location and science variables for the GEDI shots using OPeNDAP.\nFor this tutorial, we will use the boundary of Annapurna Conservation Area (ACA), the largest protected area of Nepal, to search for the overlapping GEDI L4A granules. The boundary polygon is available as a GeoJSON file at aca.json. Let’s plot the boundary polygon of the ACA over a basemap.\n\n# read and plot ACA GeoJSON file\naca = gpd.read_file(\"https://raw.githubusercontent.com/ornldaac/gedi_tutorials/main/polygons/aca.json\") \nax=aca.to_crs(epsg=3857).plot(figsize=(5, 6), edgecolor=\"red\", alpha=0.3)\nctx.add_basemap(ax)\nax.set_title(\"Annapurna Conservation Area, Nepal\")\nplt.show()\n\n\n\n\nWe will now search for all the GEDI L4A granules overlapping the ACA boundary polygon using the Earthaccess Python library. We will also limit the period of our search to the year 2020 (Jan 1 to Dec 31, 2020). The earthaccess’s search_data allows searching dataset granules using NASA’s CMR API. It requires the dataset digital object identifier (DOI), time, and the polygon coordinates as tuples.\n\n# GEDI L4A DOI \ndoi = '10.3334/ORNLDAAC/2056'\n\n# time-bound for the year 2020\nstart_date = \"2020-01-01\" # specify your start date\nend_date = \"2020-12-31\" # specify your end date\n\n# bounding lon, lat as a list of tuples\naca.geometry = aca.geometry.apply(orient, args=(1,))\n# simplifying the polygon to bypass the coordinates \n# limit of the CMR with a tolerance of .005 degrees\nxy = aca.geometry.simplify(0.005).get_coordinates()\ncoord_list = [(x,y) for x,y in zip(xy.x , xy.y)]\n\ngranule_arr = earthaccess.search_data(\n count=-1, # needed to retrieve all granules\n doi=doi,\n polygon=coord_list,\n temporal=(start_date, end_date)\n)\n\nGranules found: 59\n\n\nNow, the GEDI L4A granule information, including data URLs, is stored in granule_arr, from which we can create OPeNDAP URLs. We will use NetCDF-4 encoding in the HTTP requests as an NC4 file representation. Let’s print the first three granules from the list.\n\n# create opendap urls from the granule links\nopendap_base ='https://opendap.earthdata.nasa.gov'\nopendap_url = f'{opendap_base}/collections/C2237824918-ORNL_CLOUD/granules/GEDI_L4A_AGB_Density_V2_1'\nopendap_arr = [f'{opendap_url}.{path.basename(granule.data_links()[0])}.dap.nc4' for granule in granule_arr]\ntotal_granules = len(opendap_arr)\n# print the first three links\nopendap_arr[:3]\n\n['https://opendap.earthdata.nasa.gov/collections/C2237824918-ORNL_CLOUD/granules/GEDI_L4A_AGB_Density_V2_1.GEDI04_A_2020009160947_O06097_02_T00100_02_002_02_V002.h5.dap.nc4',\n 'https://opendap.earthdata.nasa.gov/collections/C2237824918-ORNL_CLOUD/granules/GEDI_L4A_AGB_Density_V2_1.GEDI04_A_2020013143447_O06158_02_T04675_02_002_02_V002.h5.dap.nc4',\n 'https://opendap.earthdata.nasa.gov/collections/C2237824918-ORNL_CLOUD/granules/GEDI_L4A_AGB_Density_V2_1.GEDI04_A_2020024182123_O06331_03_T00197_02_002_02_V002.h5.dap.nc4']" }, { - "objectID": "how-tos/using-s3-storage.html#move-existing-data-from-home-to-persistent_bucket", - "href": "how-tos/using-s3-storage.html#move-existing-data-from-home-to-persistent_bucket", - "title": "Using S3 Bucket Storage in NASA-Openscapes Hub", - "section": "", - "text": "You may already have some data in your HOME directory that you would like to move out to a persistent bucket. You can do that using the awsv2 s3 command line tool, which is already installed on the hub. You can open a terminal from the Hub Launcher - it will open in your HOME directory. You can then use the awsv2 s3 mv command to move a file to your bucket.\n\n\n$ awsv2 s3 mv my-big-file.nc $PERSISTENT_BUCKET/ # The trailing slash is important here\nmove: ./my-big-file.nc to s3://openscapeshub-persistent/ateucher/my-big-file.nc\n\n\n\nList the contents of the local results-data directory:\n$ ls results-data/\nmy-big-file1.nc my-big-file2.nc\nUse awsv2 s3 mv with the --recursive flag to move all files in a directory to a new directory in PERSISTENT_BUCKET\n$ awsv2 s3 mv --recursive results-data $PERSISTENT_BUCKET/results-data/\nmove: results-data/my-big-file1.nc to s3://openscapeshub-persistent/ateucher/results-data/my-big-file1.nc\nmove: results-data/my-big-file2.nc to s3://openscapeshub-persistent/ateucher/results-data/my-big-file2.nc" + "objectID": "tutorials/Above_Ground_Biomass_Density.html#access-the-variables-of-interest-for-the-subset-area", + "href": "tutorials/Above_Ground_Biomass_Density.html#access-the-variables-of-interest-for-the-subset-area", + "title": "Distribution of carbon stocks in a protected area", + "section": "Access the variables of interest for the subset area", + "text": "Access the variables of interest for the subset area\nOPeNDAP allows access to the dataset of interest, so you can only download the variables you are interested in and, by doing so, save the download bandwidth. For this tutorial, we are interested in the following selected science variables (agbd, l4_quality_flag, and pft_class) for four full-powered beams of the eight GEDI beams. Refer to this tutorial for more information about GEDI L4A data structure.\n\n# variables of interests\nvariables = ['agbd', 'l4_quality_flag', 'land_cover_data/pft_class']\n\n# beams of interest\nbeams = ['BEAM0101', 'BEAM0110', 'BEAM1000', 'BEAM1011']\n\n# setting header variables\nheaders = ['lat_lowestmode', 'lon_lowestmode', 'elev_lowestmode', 'shot_number']\nheaders.extend(variables)\n\nNow, we will retrieve the dataset from the OPeNDAP Hyrax server. Let’s set up a session and set the number of maximum retries.\n\n# setting maximum retries for the HTTP sessions to get around the 500 error\nsession = earthaccess.get_requests_https_session()\nretries = Retry(total=10, backoff_factor=0.1, status_forcelist=[ 500, 502, 503, 504 ])\nsession.mount('https://', HTTPAdapter(max_retries=retries))\n\nFirst, we will access the geographic variables (lat_lowestmode, lon_lowestmode) of all the GEDI shots within the beam to identify the shots overlapping the area of interest. We will then retrieve the science variables of interest (agbd, agbd_t, etc.) for the area of interest.\nThe following step will take some time to run as there are 59 granules to process. The figure below will update to indicate the download status.\n\n# plotting the area of interest\nax=aca.to_crs(epsg=3857).plot(figsize=(5, 6), alpha=0.3)\nctx.add_basemap(ax)\nax.set_title(\"Starting download...\")\ndisplay.display(plt.gcf())\ndisplay.clear_output(wait=True)\n\nc=0\ndfs=[]\n# loop through all granule URLs\nfor g_name in opendap_arr:\n c += 1\n # loop through all beams\n for beam in beams: \n # 1. Retrieving coordinates for the beam\n r = session.get(f\"{g_name}?dap4.ce=/{beam}/lon_lowestmode;/{beam}/lat_lowestmode\")\n if (r.status_code != 400):\n with nc.Dataset('hyrax', memory=r.content) as ds:\n df = pd.DataFrame({'lat_lowestmode': ds[beam]['lat_lowestmode'][:], \n 'lon_lowestmode': ds[beam]['lon_lowestmode'][:]}) # creating pandas dataframe \n\n # 2. Subsetting by bounds of the area of interest\n # converting to geopandas dataframe\n gdf = gpd.GeoDataFrame(df, geometry=gpd.points_from_xy(df.lon_lowestmode, df.lat_lowestmode)) \n gdf_aca = gdf[gdf['geometry'].within(aca.geometry[0])] \n if not gdf_aca.empty:\n # creating empty dataframe columns for variables\n for v in headers[2:]:\n gdf_aca[v] = pd.Series()\n # 3. Retrieving variables of interest. Only retrieving GEDI shots within the subset area.\n for _, df_gr in gdf_aca.groupby((gdf_aca.index.to_series().diff() > 1).cumsum()):\n i = df_gr.index.min()\n j = df_gr.index.max()\n for v in headers[2:]:\n r = session.get(f\"{g_name}?dap4.ce=/{beam}/{v}%5B{i}:{j}%5D\")\n if (r.status_code != 400):\n with nc.Dataset('hyrax', memory=r.content) as ds:\n gdf_aca.loc[i:j, (v)] = ds[beam][v][:]\n\n # creating a dask dataframe\n dfs.append(dd.from_pandas(pd.DataFrame(gdf_aca.drop(columns='geometry')), npartitions=3))\n\n # plotting the GEDI shots on map\n gdf_aca.crs = \"EPSG:4326\"\n gdf_aca.to_crs(epsg=3857).plot(alpha=0.01, ax=ax, linewidth=0)\n ax.set_title(f\"Downloading {c} of {total_granules}... {g_name.rsplit('/', 1)[-1][:-8]} / {beam}\")\n display.display(plt.gcf())\n display.clear_output(wait=True)\n\n\n\n\nLet’s concatenate the dataframes into a single pandas dataframe and print the first few rows. The concatenated dataframe now contains the subset data for the GEDI L4A shots within the ACA boundary, which can also be saved as a CSV file, as shown below (commented out).\n\n# concating the dataframe \nl4a_df = dd.concat(dfs).compute()\n# uncomment the line below to output the data as a CSV file\n# l4a_df.to_csv('aca_gedi_l4a.csv', index=False, header=False, columns=headers)\n# printing the first few rows\nl4a_df.head()\n\n\n\n\n\n\n\n\nlat_lowestmode\nlon_lowestmode\nelev_lowestmode\nshot_number\nagbd\nl4_quality_flag\nland_cover_data/pft_class\n\n\n\n\n69740\n28.533718\n83.651917\n1334.218384\n60970500200160455\n2.014405\n0\n6\n\n\n69741\n28.534097\n83.652311\n1336.080078\n60970500200160456\n63.936985\n0\n4\n\n\n69742\n28.534474\n83.652703\n1344.832642\n60970500200160457\n18.22143\n0\n4\n\n\n69743\n28.534852\n83.653096\n1349.221069\n60970500200160458\n76.016953\n0\n4\n\n\n69744\n28.535227\n83.653488\n1362.543945\n60970500200160459\n23.200373\n0\n4" }, { - "objectID": "how-tos/read_data.html", - "href": "how-tos/read_data.html", - "title": "Read Data", + "objectID": "tutorials/Above_Ground_Biomass_Density.html#distribution-of-agbd-in-annapurna-conservation-area", + "href": "tutorials/Above_Ground_Biomass_Density.html#distribution-of-agbd-in-annapurna-conservation-area", + "title": "Distribution of carbon stocks in a protected area", + "section": "Distribution of AGBD in Annapurna Conservation Area", + "text": "Distribution of AGBD in Annapurna Conservation Area\nThe pft_class variable provides the plant functional types for the shots derived from the MODIS land cover type product (MCD12Q1). It uses LC_Type5 or Annual PFT classification. We can now plot the distribution of the AGBD by plant functional types (PFTs) for the good-quality shots in the ACA region.\n\n# MCD12Q1 PFT types\npft_legend = {0 : 'Water Bodies', 1: 'Evergreen Needleleaf Trees', 2: 'Evergreen Broadleaf Trees', \n 3: 'Deciduous Needleleaf Trees', 4: 'Deciduous Broadleaf Trees', 5: 'Shrub', 6: 'Grass',\n 7: 'Cereal Croplands', 8: 'Broadleaf Croplands', 9: 'Urban and Built-up Lands', \n 10: 'Permanent Snow and Ice', 11: 'Barren', 255: 'Unclassified'}\n\n# creating a mask with good quality shots and trees/shrubs PFT class\nl4a_df_masked = l4a_df[(l4a_df['l4_quality_flag']==1) & (l4a_df['land_cover_data/pft_class'] <= 5 )]\nplt.figure(figsize=(15,5))\n# drawing distribution plots\nax = l4a_df_masked.groupby('land_cover_data/pft_class')['agbd'].\\\n apply(lambda x: sns.distplot(x, label = pft_legend[x.name], hist=True))\nplt.xlabel('agbd (Mg / ha)')\nplt.title(f'Distribution of GEDI L4A AGBD estimates by PFTs (Plant Functional Types) in ACA in 2020')\nplt.legend()\nplt.show()\n\n\n\n\nLet’s plot how the AGBD is distributed across elevation ranges for different PFTs.\n\n# create 500m elevation bins\nl4a_df_masked['elev_bin']=pd.cut(l4a_df_masked['elev_lowestmode'], bins =range(0, 5000, 500))\n# drawing categorical plot\ng = sns.catplot(x = \"elev_bin\", y = \"agbd\", data = l4a_df_masked, col=\"land_cover_data/pft_class\", kind=\"box\")\ng.set_xticklabels(rotation=90)\ng.set_titles(\"{col_name}\")\nfor ax in g.axes.flat:\n ax.set_title(pft_legend[int(float(ax.get_title()))])\ng.set_axis_labels(\"Elevation (m)\")\nsns.despine(top=False, right=False, left=False, bottom=False, offset=None, trim=False)\nplt.show()\n\n\n\n\nAdditional tutorials on discovering, accessing, and using GEDI Level 3 and Level 4 data products are available at https://github.com/ornldaac/gedi_tutorials." + }, + { + "objectID": "glossary.html", + "href": "glossary.html", + "title": "Glossary, Cheatsheets, & Guides", "section": "", - "text": "PythonRMATLAB\n\n\nWe recommend using xarray’s open_dataset\n\n\nimport xarray as xr\nds = xr.open_dataset(filename)\n\n\n\nimport xarray as xr\nds = xr.open_mfdataset(list_of_filenames)\n\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB provides graphical import tools and two programmatic interfaces for interacting with netCDF files.\nThe high-level interface is made of eight high-level functions for getting information about netCDF files and their content, creating variables, writing and reading variables and attributes, and adding schema definitions to netCDF files.\nfilename = \"example.nc\";\nncdisp(filename); % Display file content\nvarname = \"peaks\";\npeaksData = ncread(filename,varname);\nsurf(peaksData);\nThe low-level interface is a thin wrapper around the underlying netCDF C library.\nfilename = \"example.nc\";\nvarname = \"peaks\";\nncID = netcdf.open(filename,\"NC_NOWRITE\");\nvarID = netcdf.inqVarID(ncID,varname);\npeaksData = netcdf.getVar(ncID,varID);\nnetcdf.close(ncID);\nsurf(peaksData);\nNotes\n\nUnlike MATLAB HDF5 interfaces, netCDF interfaces are not cloud enabled (yet).\nAs HDF5 interfaces support reading netCDF 4 files, users must currently use these interfaces for accessing netCDF 4 files stored on the cloud (see tutorial MATLAB Access Single NASA EarthData L2 NetCDF in this Cookbook).\nMATLAB netCDF interfaces support OPeNDAP and byte-range for reading data." + "text": "A new paradigm brings new terminology you will encounter while you learn about cloud data access and computing.\nThis handy cheatsheet, Cloud Terminology 101, defines commonly used cloud computing terms and phrases.\n\n\nPublished Google Slide\nMore specific to working with NASA Earthdata in the cloud, this Workflow Terminology Cheatsheet defines terms associated with NASA Earthdata, cloud-optimized data, and some open software science tools you may encounter. A full list of NASA Eathdata-specific terms are found in the NASA Earthdata Glossary.\n\n\nPublished Google Slide" }, { - "objectID": "how-tos/read_data.html#netcdf", - "href": "how-tos/read_data.html#netcdf", - "title": "Read Data", + "objectID": "glossary.html#glossary", + "href": "glossary.html#glossary", + "title": "Glossary, Cheatsheets, & Guides", "section": "", - "text": "PythonRMATLAB\n\n\nWe recommend using xarray’s open_dataset\n\n\nimport xarray as xr\nds = xr.open_dataset(filename)\n\n\n\nimport xarray as xr\nds = xr.open_mfdataset(list_of_filenames)\n\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB provides graphical import tools and two programmatic interfaces for interacting with netCDF files.\nThe high-level interface is made of eight high-level functions for getting information about netCDF files and their content, creating variables, writing and reading variables and attributes, and adding schema definitions to netCDF files.\nfilename = \"example.nc\";\nncdisp(filename); % Display file content\nvarname = \"peaks\";\npeaksData = ncread(filename,varname);\nsurf(peaksData);\nThe low-level interface is a thin wrapper around the underlying netCDF C library.\nfilename = \"example.nc\";\nvarname = \"peaks\";\nncID = netcdf.open(filename,\"NC_NOWRITE\");\nvarID = netcdf.inqVarID(ncID,varname);\npeaksData = netcdf.getVar(ncID,varID);\nnetcdf.close(ncID);\nsurf(peaksData);\nNotes\n\nUnlike MATLAB HDF5 interfaces, netCDF interfaces are not cloud enabled (yet).\nAs HDF5 interfaces support reading netCDF 4 files, users must currently use these interfaces for accessing netCDF 4 files stored on the cloud (see tutorial MATLAB Access Single NASA EarthData L2 NetCDF in this Cookbook).\nMATLAB netCDF interfaces support OPeNDAP and byte-range for reading data." + "text": "A new paradigm brings new terminology you will encounter while you learn about cloud data access and computing.\nThis handy cheatsheet, Cloud Terminology 101, defines commonly used cloud computing terms and phrases.\n\n\nPublished Google Slide\nMore specific to working with NASA Earthdata in the cloud, this Workflow Terminology Cheatsheet defines terms associated with NASA Earthdata, cloud-optimized data, and some open software science tools you may encounter. A full list of NASA Eathdata-specific terms are found in the NASA Earthdata Glossary.\n\n\nPublished Google Slide" }, { - "objectID": "how-tos/read_data.html#hdf5", - "href": "how-tos/read_data.html#hdf5", - "title": "Read Data", - "section": "HDF5", - "text": "HDF5\n\nPythonRMATLAB\n\n\nFor HDF5 files, there are two methods we can use: xarray’s open_dataset and h5py\nimport xarray as xr\nds = xr.open_dataset(filename,\n group=path_to_H5netCDF_group)\n# add in directions for h5py\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB provides low- and high-level programmatic interfaces for interacting with HDF5 and HDF4 files.\nFocusing on HDF5, which is currently more prevalent, the high-level interface is made of seven high-level functions for getting information about HDF5 files and their content, creating datasets, and reading and writing datasets and attributes.\nfilename = \"example.h5\";\nh5disp(filename); % Display file content\nds = \"/g4/lat\";\nlat = h5read(filename,ds);\nThe low-level interface is a thin wrapper around the underlying HDF5 C library. It is made of more than 300 functions organized in packages (H5F for file management, H5D for dataset management, etc.).\nfilename = \"example.h5\";\nds = \"/g4/lat\";\nfID = H5F.open(filename); \ndsID = H5D.open(fID,ds);\nlat = H5D.read(dsID);\nH5D.close(dsID);\nH5F.close(fID);" + "objectID": "glossary.html#cheatsheets-guides", + "href": "glossary.html#cheatsheets-guides", + "title": "Glossary, Cheatsheets, & Guides", + "section": "Cheatsheets & Guides", + "text": "Cheatsheets & Guides\nHow are all of these terms and concepts related? And how can you begin to put them to work?\nHere are some cheatsheets and guides to help visualize what working with NASA Earthdata in the cloud can look like, and how to get started.\nAll slides and cheatsheets are available for re-use and remix! Let us know what you think! We welcome your input so we can continue to improve and update these guides. Slides are credited for each deck; Cheatsheets development have been led by Catalina Oaida Taglialatela and Cassie Nickles (PO.DAAC) in Spring 2022.\n(Internal link).\n\nWhat is the NASA Earthdata Cloud?\nNASA Earthdata Cloud is the NASA archive of Earth observations and is hosted in Amazon Web Services (AWS) cloud with DAAC tools and services built for use “next to the data.” The NASA DAACs (data centers) are currently transitioning to this cloud-based environment. The cloud offers a scalable and effective way to address storage, network, and data movement concerns while offering a tremendous amount of flexibility to the user. Particularly if working with large data volumes, data access and processing would be more efficient if workflows are taking place in the cloud, avoiding having to download large data volumes. Data download will continue to be freely available to users, from the Earthdata Cloud archive.\n\n\nPublished Google Slide\n\n\nCloud Access Pathways\nThree pathway examples, to interact and access data (and services) from and within the NASA Earthdata Cloud, are illustrated in the diagram. Green arrows and icons indicate working locally, after downloading data to your local machine, servers, or compute/storage space. Orange arrows and icons highlight a workflow within the cloud, setting up your own AWS EC2 cloud instance, or virtual machine, in the cloud next to the data. Blue arrows and icons also indicate a within the cloud workflow, through shareable cloud environments such as Binder or JupyterHub set up in an AWS cloud region. Note that each of these may have a range of cost models. EOSDIS data are being stored in the us-west-2 region of AWS cloud; we recommend setting up your cloud computing environment in the same region as the data for free and easy in-cloud access.\n\n\nPublished Google Slide\nA note on costing: What is free and what do I have to budget for, now that data is archived in the cloud?\n\nDownloading data from the Earthdata Cloud archive in AWS, to your local computer environment or local storage (e.g. servers) is and will continue to be free for the user.\nAccessing the data directly in the cloud (from us-west-2 S3 region) is free. Users will need a NASA Earthdata Login account and AWS credentials to access, but there is no cost associated with these authentication steps, which are in place for security reasons.\nAccessing data in the cloud via EOSDIS or DAAC cloud-based tools and services such as the CMR API, Harmony API, OPenDAP API (from us-west-2 S3 region) is free to the user. Having the tools and services “next to the data” in the cloud enables DAACs to support data reduction and transformation, more efficiently, on behalf of the user, so users only access the data they need.\nCloud computing environments (i.e. virtual machines in the cloud) for working with data in the cloud (beyond direct or via services provided access) such as data analysis or running models with the data, is user responsibility, and should be considered in budgeting. I.e. User would need to set up a cloud compute environment (such as an EC2 instance or JupyterLab) and are responsible for any storage and computing costs.\n\nThis means that even though direct data access in the cloud is free to the user, they would first need to have a cloud computing environment/machine to execute the data access step from, and then continue their analysis.\nDepending on whether that cloud environment is provided by the user themselves, user’s institution, community hubs like Pangeo or NASA Openscapes JupyterLab sandbox, this element of the workflow may require user accountability, budgeting and user financial maintenance.\n\n\n\n\nGetting Started Roadmap\n\nCloud Workflow\nThe following is a conceptual roadmap for users getting started with NASA Earth Observations cloud-archived data using an in-cloud workflow (i.e. bringing user code into the cloud, avoiding data download and performing data workflows “next to the data”).\n\n\nPublished Google Slide\n\n\nLocal Workflow\nThe following is a conceptual roadmap for users getting started with NASA Earth Observations cloud-archived data using a local machine (e.g. laptop) workflow, as data storage and computational work.\n\n\nPublished Google Slide\n\n\n\nTools & Services Roadmap\nBelow is a practical guide for learning about and selecting helpful tools or services for a given use case, focusing on how to find and access NASA Earthdata Cloud-archived data from local compute environment (e.g. laptop) or from a cloud computing workspace, with accompanying example tutorials. Once you follow your desired pathway, click on the respective blue notebook icon to get to the example tutorial. Note: these pathways are not exhaustive, there are many ways to accomplish these common steps, but these are some of our recommendations.\n\n\nPublished Google Slide\n\n\nWorkflow Cheatsheet\nThe following is a practical reference guide with links to tutorials and informational websites for users who are starting to take the conceptual pieces and explore and implement in their own workflows.\n\n\nPublished Google Slide" }, { - "objectID": "how-tos/read_data.html#geotiff", - "href": "how-tos/read_data.html#geotiff", - "title": "Read Data", - "section": "GeoTIFF", - "text": "GeoTIFF\n\nPythonRMATLAB\n\n\nFor GeoTIFF files, we recommend using rioxarray’s open_rasterio\nimport rioxarray\nxds = rioxarray.open_rasterio(\"my.tif\")\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB Mapping Toolbox supports interacting with GeoTIFF files through a series of functions for handling raster data (under Raster Data), and the function geotiffread althrough not recommended but for a few edge cases.\n[A,R] = readgeoraster(\"myGeoTiffFile.tif\");\nmapshow(A,R);" + "objectID": "glossary.html#slides", + "href": "glossary.html#slides", + "title": "Glossary, Cheatsheets, & Guides", + "section": "Slides", + "text": "Slides\nSelected presentations about working with NASA Earthdata on the Cloud; for all presentations see nasa-openscapes.github.io > slides.\n\nNASA Earthdata Cloud: Myths, Truths, Questions\nby Amy Steiker, Kate Heightley (NSIDC) September 7, 2022\n\n\nNSIDC DAAC User Working Group\nby Andrew Barrett, Amy Steiker, Walt Meier, Jennie Roebuck, Mikala Beig, Luis Lopez, (NSIDC) May 20, 2022.\n\n\nNASA Earthdata Cloud & The Cloud Paradigm\nby Aaron Friesz (LP DAAC), April 2022." }, { - "objectID": "how-tos/read_data.html#shapefiles-geojson", - "href": "how-tos/read_data.html#shapefiles-geojson", - "title": "Read Data", - "section": "Shapefiles & GeoJSON", - "text": "Shapefiles & GeoJSON\n\nPythonRMATLAB\n\n\nTo open shapefiles or GeoJSON, we recommend using geopandas’s read_file.\nimport geopandas as gpd\ngdf = gpd.read_file(filename)\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB Mapping Toolbox supports interacting with shapefiles and GeoJSON files through a series of functions for handling vector data (under Vector Data), in particular readgeotable, shapeinfo, shaperead, and shapewrite.\n% Read shapefile of countries specifying the name of the attributes to read.\nS = shaperead(\"Countries.shp\",\"Attributes\",[\"ISO_2DIGIT\",\"CNTRY_NAME\"]);\n\n% Read GeoJSON data from a website and save it in the file storms.geojson.\nwebsave(\"storms.geojson\",\"https://www.spc.noaa.gov/products/outlook/day1otlk_cat.lyr.geojson\");\nGT = readgeotable(\"storms.geojson\");" + "objectID": "appendix/index.html", + "href": "appendix/index.html", + "title": "Appendix", + "section": "", + "text": "Extra resources for Earthdata Cloud." }, { - "objectID": "how-tos/read_data.html#csv", - "href": "how-tos/read_data.html#csv", - "title": "Read Data", - "section": "CSV", - "text": "CSV\n\nPythonRMATLAB\n\n\nTo open CSV files, we recommend using pandas’s read_csv.\nimport pandas as pd\ndf = pd.read_csv(filename)\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB supports interacting with CSV files and other tabular file formats through functions readtable, readtimetable, readcell, and readmatrix, as well as their counterparts for writing to files.\n% Read time table and display summary\nTT = readtimetable(\"myTimeTable.csv\");\nsummary(TT)\nAdvanced workflows can be implemented using detectImportOptions and functions referenced in its documentation.\n% Read variables \"TaxiIn\" and \"TaxiOut\" from airlines dataset\nfilename = \"airlinesmall.csv\";\nopts = detectImportOptions(filename);\nopts.SelectedVariableNames = [\"TaxiIn\",\"TaxiOut\"];\nT = readtable(filename, opts);\nNotes\n\nMATLAB provides higher-level mechanisms form importing data files through the function importdata and the Import Tool.\nA general list of functions and workflows for interacting with deliimited and formatted text files is provided here." + "objectID": "appendix/index.html#about-the-cloud", + "href": "appendix/index.html#about-the-cloud", + "title": "Appendix", + "section": "About the Cloud", + "text": "About the Cloud\nMaybe you’ve heard that NASA Earthdata is “moving to the cloud” but you want to know why. You can read the details of the Earthdata Cloud Evolution, but here we summarize the benefits of the cloud and additional resources on its use and history. In short, the cloud will make the download of data unnecessary, allow for data processing and manipulation without new software purchases or installation, and, ultimately, reduce the amount of time it takes to get the data needed to do science.\n\nAmazon Web Services\nNASA’s Office of the Chief Information Officer chose Amazon Web Services (AWS) as the source of general-purpose cloud services (but some areas within NASA are working with Google Earth Engine (GEE) to make NASA data accessible in the GEE cloud-based analysis platform). The following resources provide a background on AWS, but much of the information is relevant to folks who want to develop in the cloud rather than simply access data. Remember, all NASA’s science information (including the algorithms, metadata, and documentation associated with science mission data) must be freely available to the public. This means that anyone, anywhere in the world, can access NASA Earth science data without restriction. However, advanced cloud operations could require a user to set-up their own cloud account through AWS or another cloud provider.\n\nCloud Primer for Amazon Web Services This primer provides step-by-step tutorials on how to get started in the AWS cloud.\nWhat is AWS Amazon Web Services is the world’s most comprehensive and broadly adopted cloud, offering over 200 fully featured services from data centers globally.\n\n\n\nCloud Optimized Data Formats\nTraditional file formats can easily be migrated to the cloud, but serving or processing the data from the cloud is inefficient and often requires that the data be downloaded and then translated to another format and stored in memory. Cloud optimized formats are being developed to better serve analysis-in-place workflows that make the cloud so beneficial to science users.\n\nCloud-Optimized Format Study The cloud infrastructure provides a number of capabilities that can dramatically improve access and use of Earth Observation data. However, in many cases, data may need to be reorganized and/or reformatted in order to make them tractable to support cloud-native analysis and access patterns. The purpose of this study is to examine different formats for storing data on the cloud.\nCloud Optimized GeoTIFF A Cloud Optimized GeoTIFF is a regular GeoTIFF file with an internal organization that enables more efficient workflows on the cloud. It does this by leveraging the ability of clients issuing ​HTTP GET range requests to ask for just the parts of a file they need.\nCloud Optimized Formats: NetCDF-as-Zarr Optimizations and Next Steps Building on the work by USGS/HDF to access netCDF as Zarr, the authors found that a sidecar metadata record that includes byte offsets provides users “access HDF5 format data as efficiently as Zarr format data using the Zarr library.” In other words, users can gain the cloud-optimized performance of Zarr while retaining the archival benefits of NetCDF4." }, { - "objectID": "how-tos/read_data.html#excel", - "href": "how-tos/read_data.html#excel", - "title": "Read Data", - "section": "Excel", - "text": "Excel\n\nPythonRMATLAB\n\n\nTo open Excel files, we recommend using pandas’s read_excel\nimport pandas as pd\ndf = pd.read_excel(filename)\n\n\nR code coming soon!\n# Coming soon!\n\n\nSee section about CSV files." + "objectID": "appendix/index.html#environments", + "href": "appendix/index.html#environments", + "title": "Appendix", + "section": "Environments", + "text": "Environments\n\nManaging Python Environments This book is intended to introduce students to modern computing software, programming tools, and best practices that are broadly applicable to the analysis and visualization of Earth and Environmental data. This section describes basic programming in the open-source Python language.\nReproducible and upgradable Conda environments with conda-lock\nManaging virtual environments with pyenv\nUnderstanding and Improving Conda’s performance\nThe definitive guide to Python virtual environments with conda\nMaking conda fast again" }, { - "objectID": "how-tos/read_data.html#mat-files-.mat", - "href": "how-tos/read_data.html#mat-files-.mat", - "title": "Read Data", - "section": "MAT-Files (.mat)", - "text": "MAT-Files (.mat)\n\nPythonRMATLAB\n\n\n#coming soon! scipy & .mat\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB can interact with MAT-Files (.mat) using functions load, save, and through matfile objects.\nNotes\n\nMATLAB provides higher-level mechanisms form importing data files through the function importdata and the Import Tool.\nMultiple versions of the MAT-File format have been released. The latest, MAT-File v7.3, is based on the HDF5 file format." + "objectID": "appendix/index.html#development-on-aws", + "href": "appendix/index.html#development-on-aws", + "title": "Appendix", + "section": "Development on AWS", + "text": "Development on AWS\n\nPython\n\nPython on AWS Tools, docs, and sample code to develop applications on the AWS cloud.\n\n\n\nR\n\nGetting started with R on Amazon Web Services This guide demonstrates how to use AWS in R with the Paws AWS software development kit.\nR for Cloud Computing This book will help you kick-start analytics on the cloud including chapters on both cloud computing, R, common tasks performed in analytics." }, { - "objectID": "how-tos/read_data.html#text-data-files", - "href": "how-tos/read_data.html#text-data-files", - "title": "Read Data", - "section": "Text Data Files", - "text": "Text Data Files\n\nPythonRMATLAB\n\n\n# Coming soon!\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB provides a series of functions for interacting with text files. In addition to functions listed in sections about CSV and Binary files, MATLAB provides functions fileread for reading the content of a whole file as text, and readlines and writelines for reading/writing lines to text files.\nNotes\n\nMATLAB provides higher-level mechanisms form importing data files through the function importdata and the Import Tool.\nMATLAB patterns and regular expressions are useful for processing unstructured text data." + "objectID": "appendix/index.html#additional-coding-resources", + "href": "appendix/index.html#additional-coding-resources", + "title": "Appendix", + "section": "Additional Coding Resources", + "text": "Additional Coding Resources\n\nPython\n\nIntro to Geospatial Raster and Vector Data with Python This tutorial provides an introduction to raster data, and describes how to plot, program, and access satellite imagery using Python.\n\n\n\nR\n\nR for Data Science Online Learning Community The R4DS Online Learning Community is a community of R learners at all skill levels working together to improve their skills.\nThe Environmental Data Science Book This book is a living, open and community-driven online resource to showcase and support the publication of data, research and open-source tools for collaborative, reproducible and transparent Environmental Data Science.\nCU EarthLab’s Earth Data Science This site offers free online courses, tutorials, and tools for earth science using R and Python." }, { - "objectID": "how-tos/read_data.html#binary-files", - "href": "how-tos/read_data.html#binary-files", - "title": "Read Data", - "section": "Binary Files", - "text": "Binary Files\n\nPythonRMATLAB\n\n\nTo open binary files, we recommend using numpy’s from_file. You will need to know the dimensions and data type of the binary file and if there is an offset or header information in the binary file.\nimport numpy as np\narr = np.from_file(filepath, dtype=data_type).reshape(nrows, ncols)\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB supports interacting with binary files through low-level IO functions fopen, fclose, ferror, fseek, ftell, feof, fscanf, fprintf, fread, fwrite, and frewind.\nNotes\n\nMATLAB provides higher-level mechanisms form importing data files through the function importdata and the Import Tool.\nIt is rare that ones must implement custom binary data file readers/writers, because MATLAB and its toolboxes provide functions for reading and writing a large number of data file formats." + "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html", + "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html", + "title": "Accessing Cloud Optimized GeoTIFF (COG) - HTTPS Example", + "section": "", + "text": "In this notebook, we will access data for the Harmonized Landsat Sentinel-2 (HLS) Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0 (L30) ([10.5067/HLS/HLSL30.002](https://doi.org/10.5067/HLS/HLSL30.002)) data product. These data are archived and distributed as Cloud Optimized GeoTIFF (COG) files, one file for each spectral band.\nWe will access a single COG file, L30 red band (0.64 – 0.67 μm), from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataarray. This approach leverages S3 native protocols for efficient access to the data." }, { - "objectID": "how-tos/find-data/programmatic.html", - "href": "how-tos/find-data/programmatic.html", - "title": "How do I find data using code?", + "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#summary", + "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#summary", + "title": "Accessing Cloud Optimized GeoTIFF (COG) - HTTPS Example", "section": "", - "text": "Here are our recommended approaches for finding data with code, from the command line or a notebook.\n\nPythonRMatlabCommand Line\n\n\nIn Python we can use the earthaccess library (renamed, previously earthdata)\nTo install the package we’ll run this code from the command line. Note: you can run shell code directly from a Jupyter Notebook cell by adding a !, so it would be !conda install.\n## In the command line\n## Install earthaccess\nconda install -c conda-forge earthaccess\nThis example searches for data from the Land Processes DAAC with a spatial bounding box and temporal range.\n## In Python\n## Import packages\nfrom earthaccess import DataGranules, DataCollections\nfrom pprint import pprint \n\n\n## We'll get 4 collections that match with our keyword of interest\ncollections = DataCollections().keyword(\"REFLECTANCE\").cloud_hosted(True).get(4)\n\n## Let's print 2 collections\nfor collection in collections[0:2]:\n print(pprint(collection.summary()) , collection.abstract(), \"\\n\")\n \n## Search for files from the second dataset result over a small plot in Nebraska, USA for two weeks in September 2022\ngranules = DataGranules().concept_id(\"C2021957657-LPCLOUD\").temporal(\"2022-09-10\",\"2022-09-24\").bounding_box(-101.67271,41.04754,-101.65344,41.06213)\nprint(len(granules))\ngranules\n\n\nTo find data in R, we’ll also use the earthaccess python package - we can do so from R using the reticulate package (cheatsheet). Note below that we import the python library as an R object we name earthaccess, as well as the earthaccess$ syntax for accessing functions from the earthaccess library. The granules object has a list of JSON dictionaries with some extra dictionaries.\n## In R\n## load R libraries\nlibrary(tidyverse) # install.packages(\"tidyverse\") \nlibrary(reticulate) # install.packages(\"reticulate\")\n\n## load python library\nearthaccess <- reticulate::import(\"earthaccess\") \n\n## use earthaccess to access data # https://nsidc.github.io/earthaccess/tutorials/search-granules/\ngranules <- earthaccess$search_data(\n doi = \"10.5067/SLREF-CDRV3\",\n temporal = reticulate::tuple(\"2017-01\", \"2017-02\") # with an earthaccess update, this can be simply c() or list()\n)\n\n## Granules found: 72\n\n## exploring\ngranules # this is the result of the get request. \n\nclass(granules) # \"list\"\n## granules <- reticulate::py_to_r(granules) # Object to convert is not a Python object\n\n\nMatlab code coming soon!\n## In Matlab\n## Coming soon!\n\n\nWith wget and curl:\n## In the command line\n## Coming soon!" + "text": "In this notebook, we will access data for the Harmonized Landsat Sentinel-2 (HLS) Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0 (L30) ([10.5067/HLS/HLSL30.002](https://doi.org/10.5067/HLS/HLSL30.002)) data product. These data are archived and distributed as Cloud Optimized GeoTIFF (COG) files, one file for each spectral band.\nWe will access a single COG file, L30 red band (0.64 – 0.67 μm), from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataarray. This approach leverages S3 native protocols for efficient access to the data." }, { - "objectID": "how-tos/find-data/programmatic.html#introduction", - "href": "how-tos/find-data/programmatic.html#introduction", - "title": "How do I find data using code?", - "section": "", - "text": "Here are our recommended approaches for finding data with code, from the command line or a notebook.\n\nPythonRMatlabCommand Line\n\n\nIn Python we can use the earthaccess library (renamed, previously earthdata)\nTo install the package we’ll run this code from the command line. Note: you can run shell code directly from a Jupyter Notebook cell by adding a !, so it would be !conda install.\n## In the command line\n## Install earthaccess\nconda install -c conda-forge earthaccess\nThis example searches for data from the Land Processes DAAC with a spatial bounding box and temporal range.\n## In Python\n## Import packages\nfrom earthaccess import DataGranules, DataCollections\nfrom pprint import pprint \n\n\n## We'll get 4 collections that match with our keyword of interest\ncollections = DataCollections().keyword(\"REFLECTANCE\").cloud_hosted(True).get(4)\n\n## Let's print 2 collections\nfor collection in collections[0:2]:\n print(pprint(collection.summary()) , collection.abstract(), \"\\n\")\n \n## Search for files from the second dataset result over a small plot in Nebraska, USA for two weeks in September 2022\ngranules = DataGranules().concept_id(\"C2021957657-LPCLOUD\").temporal(\"2022-09-10\",\"2022-09-24\").bounding_box(-101.67271,41.04754,-101.65344,41.06213)\nprint(len(granules))\ngranules\n\n\nTo find data in R, we’ll also use the earthaccess python package - we can do so from R using the reticulate package (cheatsheet). Note below that we import the python library as an R object we name earthaccess, as well as the earthaccess$ syntax for accessing functions from the earthaccess library. The granules object has a list of JSON dictionaries with some extra dictionaries.\n## In R\n## load R libraries\nlibrary(tidyverse) # install.packages(\"tidyverse\") \nlibrary(reticulate) # install.packages(\"reticulate\")\n\n## load python library\nearthaccess <- reticulate::import(\"earthaccess\") \n\n## use earthaccess to access data # https://nsidc.github.io/earthaccess/tutorials/search-granules/\ngranules <- earthaccess$search_data(\n doi = \"10.5067/SLREF-CDRV3\",\n temporal = reticulate::tuple(\"2017-01\", \"2017-02\") # with an earthaccess update, this can be simply c() or list()\n)\n\n## Granules found: 72\n\n## exploring\ngranules # this is the result of the get request. \n\nclass(granules) # \"list\"\n## granules <- reticulate::py_to_r(granules) # Object to convert is not a Python object\n\n\nMatlab code coming soon!\n## In Matlab\n## Coming soon!\n\n\nWith wget and curl:\n## In the command line\n## Coming soon!" + "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#requirements", + "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#requirements", + "title": "Accessing Cloud Optimized GeoTIFF (COG) - HTTPS Example", + "section": "Requirements", + "text": "Requirements\n\n1. AWS instance running in us-west-2\nNASA Earthdata Cloud data in S3 can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region.\n\n\n2. Earthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\n3. netrc File\nYou will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. For additional information see: Authentication for NASA Earthdata." }, { - "objectID": "how-tos/find-data/find-r.html", - "href": "how-tos/find-data/find-r.html", - "title": "How do I find data using R?", - "section": "", - "text": "This is a work in progress, and may be split up into different modules/tutorials as we continue to work on it.\nHere are our recommended approaches for finding data with R, from the command line or a notebook.\n\nUsing the web interface\nSee How do I find data using Earthdata Search? - in that tutorial, we found the dataset ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4), with the shortname ECCO_L4_SSH_05DEG_MONTHLY_V4R4.\n\n\nSearching programmatically\nThe NASA cloud data is searchable programmatically via two methods - NASA’s own search service, and the NASA Spatio-Temporal Asset Catalog (STAC) API. To find data in R, we’ll mainly rely on the rstac package. This enables us interact with the NASA STAC API to search for our data, and at the same time learn about STAC more generally. This will be useful as it is a common standard for distributing spatial data.\nWe will also search for data using the NASA Earthdata search API, which is the service that powers the Earthdata web search portal.\nFor both of these services, the earthdatalogin package will help us get set up and provide a few handy functions to smooth some rough edges.\n\n\nAuthentication for NASA Earthdata\nAn Earthdata Login account is required to access data from the NASA Earthdata system. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\nOnce you have created your Earthdata Login account, you can use the earthdatalogin R package to help you manage your authentication within R.\nThere is some functionality currently only available in the development version of earthdatalogin, so we will install it from GitHub using the pak package:\ninstall.packages(\"pak\")\npak::pak(\"boettiger-lab/earthdatalogin\")\nThe easiest and most portable method of access is using the netrc basic authentication protocol for HTTP. Call edl_netrc() to set this up given your username and password. The easiest way to store your credentials for use in edl_netrc() is to set them as environment variables in your ~/.Renviron file. The usethis package has a handy function to find and open this file for editing:\n# install.packages(\"usethis\")\nusethis::edit_r_environ()\nThen add the following lines to the file, save the file, and restart R.\nEARTHDATA_USER=\"your_user_name\"\nEARTHDATA_PASSWORD=\"your_password\"\nNow, when you call edl_netrc(), it will consult these environment variables to create a .netrc file that will be used to authenticate with the NASA Earthdata services. If you don’t have your credentials saved as environment variables, earthdatalogin will provide its own credentials, but you may experience rate limits more readily. You can also manually type in your credentials to the username and password arguments to edl_netrc() but this is not recommended as it is too easy to accidentally share these in your code.\n\nlibrary(earthdatalogin)\n\nedl_netrc()\n\nOnce edl_netrc() has been called in your R session, then most spatial packages in R can seamlessly access NASA Earthdata over HTTP links.\n\n\nFinding data in NASA STACs with rstac\nAll of the NASA STAC catalogues can be viewed here: https://radiantearth.github.io/stac-browser/#/external/cmr.earthdata.nasa.gov/stac/.\nWe will use the rstac package to first browse the collections in the PO DAAC catalogue (POCLOUD):\n\n## In R\n## load R libraries\n# install.packages(\"pak\") \n# pak::pak(c(\"tidyverse\", \"rstac\", \"boettiger-lab/earthdatalogin\"))\n\nlibrary(rstac)\nlibrary(earthdatalogin)\n\npo_collections <- stac(\"https://cmr.earthdata.nasa.gov/stac/POCLOUD/\") |>\n collections() |>\n get_request()\n\npo_collections\n\nThis only gives us the first 10 collections in the catalogue; to get the rest we can use the collections_fetch() function:\n\nall_po_collections <- collections_fetch(po_collections)\nall_po_collections\n\nlength(all_po_collections$collections)\n\nhead(all_po_collections$collections)\n\n# Just look at the titles:\nsapply(all_po_collections$collections, `[[`, \"title\")\n\n# Get shortnames from the 'id' field and search for a match:\ngrep(\"ECCO_L4_SSH\", sapply(all_po_collections$collections, `[[`, \"id\"), value = TRUE)\n\nOnce we have searched through the collections, we can choose the one we are interested in and query it to get the items (granules) we are interested in:\n\nstart <- \"2015-01-01\"\nend <- \"2015-12-31\" \n\nitems <- stac(\"https://cmr.earthdata.nasa.gov/stac/POCLOUD\") |> \n stac_search(collections = \"ECCO_L4_SSH_05DEG_MONTHLY_V4R4\",\n datetime = paste(start,end, sep = \"/\")) |>\n post_request() |>\n items_fetch()\n\nitems\n\nThere are 13 ‘items’ representing the same 13 granules we found when we searched using EarthData Search.\nWe can see more details of the items (granules) by printing the features list item:\n\nitems$features\n\n# And the urls:\nedl_stac_urls(items)\n\n\n\nFinding data in NASA EarthData Search using earthdatalogin\nOnce we know the shortname of a collection (usually by looking in the EarthData Search portal), we can supply it to edl_search() to get the metadata and file urls of the individual granules:\n\ngranules <- edl_search(\n short_name = \"ECCO_L4_SSH_05DEG_MONTHLY_V4R4\",\n temporal = c(start, end), \n parse_results = FALSE\n)\n\ngranules\n\ngranules[[1]]\n\n# See the granule titles\nsapply(granules, `[`, \"title\")\n\n# Note these are the same urls obtained via the rstac workflow demonstrated above\ngranule_urls <- edl_extract_urls(granules)\ngranule_urls\n\n\nAccessing data using the {terra} package\nWe can read any of these urls using terra::rast(). We supply the vsi = TRUE argument, which prepends \"/vsicurl/\" to the url, indicating that the file should be opened as a “virtual” remote dataset. This allows random partial reading of files without prior download of the entire file. This will vastly speed up most operations as only the subset of the data required is ever actually downloaded.\n\nlibrary(terra)\n\nrast <- terra::rast(granule_urls[1], vsi = TRUE)\n\n# This does not come with an extent and CRS embedded so we can supply it manually\ngranules[[1]]$boxes\next(rast) <- c(-180, 180, -90, 90)\ncrs(rast) <- \"EPSG:4326\"\n\nplot(rast[[\"SSH\"]])\n\n# We notice that this plot is upside-down - it is likely that the NetCDF file\n# does not conform properly to the conventions. But we can flip it:\nrast <- flip(rast, direction = \"vertical\")\nplot(rast)\n\nIf we want to crop the raster, we can define an extent and crop it to that area. Because we previously called edl_netrc(), we are not only authenticated with the server so we can access the data, but we have also set up terra and stars to us the underlying GDAL library to access the data in such a way that only subset of the data we have requested is actually downloaded.\n\ncrop_box <- rast(extent = c(-150, -120, 35, 60), crs = \"EPSG:4326\")\n\nrast_cropped <- crop(rast, crop_box)\n\nplot(rast_cropped[[\"SSH\"]])\n\n\n\nAccessing data using the {stars} package\nThe read_* functions in stars do not have the vsi argument, but we can do the same thing simply by prepending \"/vsicurl/\" to the url ourselves. Here we will use the read_mdim() function.\n\nlibrary(stars)\nssh_stars <- read_mdim(paste0(\"/vsicurl/\", granule_urls[1]))\n\nplot(ssh_stars)\n\nWe can again crop this using the same bounding box as we did with terra:\n\nst_crop(\n ssh_stars, \n st_bbox(c(xmin = -150, xmax = -120, ymin = 35, ymax = 60))\n) |> \n plot()\n\n\n\n\nAccessing data using gdalcubes\nComing soon!" + "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#learning-objectives", + "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#learning-objectives", + "title": "Accessing Cloud Optimized GeoTIFF (COG) - HTTPS Example", + "section": "Learning Objectives", + "text": "Learning Objectives\n\nhow to configure your Python and R work environment to access Cloud Optimized geoTIFF (COG) files\nhow to access HLS COG files\nhow to plot the data" }, { - "objectID": "how-tos/reformat.html", - "href": "how-tos/reformat.html", - "title": "Reformat data files", - "section": "", - "text": "Upcoming!" + "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#cloud-optimized-geotiff-cog", + "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#cloud-optimized-geotiff-cog", + "title": "Accessing Cloud Optimized GeoTIFF (COG) - HTTPS Example", + "section": "Cloud Optimized GeoTIFF (COG)", + "text": "Cloud Optimized GeoTIFF (COG)\nUsing Harmonized Landsat Sentinel-2 (HLS) version 2.0\n\nImport Packages\n\nPythonR\n\n\nimport os\nfrom osgeo import gdal\nimport rasterio as rio\nimport rioxarray\nimport hvplot.xarray\nimport holoviews as hv\n\n\nlibrary(rgdal)\nlibrary(raster)\nlibrary(terra)" }, { - "objectID": "how-tos/reformat.html#netcdf-to-zarr", - "href": "how-tos/reformat.html#netcdf-to-zarr", - "title": "Reformat data files", - "section": "NetCDF to Zarr", - "text": "NetCDF to Zarr" + "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#workspace-environment-setup", + "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#workspace-environment-setup", + "title": "Accessing Cloud Optimized GeoTIFF (COG) - HTTPS Example", + "section": "Workspace Environment Setup", + "text": "Workspace Environment Setup\n\nPythonR\n\n\nFor this exercise, we are going to open up a context manager for the notebook using the rasterio.env module to store the required GDAL configurations we need to access the data from Earthdata Cloud. While the context manager is open (rio_env.__enter__()) we will be able to run the open or get data commands that would typically be executed within a with statement, thus allowing us to more freely interact with the data. We’ll close the context (rio_env.__exit__()) at the end of the notebook.\nGDAL environment variables must be configured to access COGs from Earthdata Cloud. Geospatial data access Python packages like rasterio and rioxarray depend on GDAL, leveraging GDAL’s “Virtual File Systems” to read remote files. GDAL has a lot of environment variables that control it’s behavior. Changing these settings can mean the difference being able to access a file or not. They can also have an impact on the performance.\nrio_env = rio.Env(GDAL_DISABLE_READDIR_ON_OPEN='TRUE',\n GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/cookies.txt'),\n GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/cookies.txt'))\nrio_env.__enter__()\n\n\nSet up rgdal configurations to access the cloud assets that we are interested in. You can learn more about these configuration options here.\nsetGDALconfig(c(\"GDAL_HTTP_UNSAFESSL=YES\",\n \"GDAL_HTTP_COOKIEFILE=.rcookies\",\n \"GDAL_HTTP_COOKIEJAR=.rcookies\",\n \"GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR\",\n \"CPL_VSIL_CURL_ALLOWED_EXTENSIONS=TIF\"))\n\n\n\nIn this example we’re interested in the HLS L30 data collection from NASA’s LP DAAC in Earthdata Cloud. Below we specify the HTTPS URL to the data asset in Earthdata Cloud. This URL can be found via Earthdata Search or programmatically through the CMR and CMR-STAC APIs.\n\nPythonR\n\n\nhttps_url = 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T11SQA.2021333T181532.v2.0/HLS.L30.T11SQA.2021333T181532.v2.0.B04.tif'\n\n\nPlease note that in R, we need to add /vsicurl/ manually to the COG file URL.\nhttps_url <- '/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T11SQA.2021333T181532.v2.0/HLS.L30.T11SQA.2021333T181532.v2.0.B04.tif'" }, { - "objectID": "how-tos/reformat.html#geotiff-to-cog", - "href": "how-tos/reformat.html#geotiff-to-cog", - "title": "Reformat data files", - "section": "GeoTIFF to COG", - "text": "GeoTIFF to COG" + "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#https-data-access", + "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html#https-data-access", + "title": "Accessing Cloud Optimized GeoTIFF (COG) - HTTPS Example", + "section": "HTTPS Data Access", + "text": "HTTPS Data Access\nRead in the HLS HTTPS URL for the L30 red band (0.64 – 0.67 μm) into our workspace. Note that, accessing files in the cloud requires you to authenticate using your NASA Earthdata Login account meaning a proper netrc file needs to be set up.\n\nPythonR\n\n\nwe are using rioxarray, an extension of xarray used to read geospatial data.\nda = rioxarray.open_rasterio(https_url)\nda\nThe file is read into Python as an xarray dataarray with a band, x, and y dimension. In this example the band dimension is meaningless, so we’ll use the squeeze() function to remove band as a dimension.\nda_red = da.squeeze('band', drop=True)\nda_red\nPlot the dataarray, representing the L30 red band, using hvplot.\nda_red.hvplot.image(x='x', y='y', cmap='gray', aspect='equal')\nExit the context manager.\nrio_env.__exit__()\n\n\nda_red <- rast(https_url)\nda_red\nThe Convert a SpatRaster object to a Raster object using raster() to be able to use leaflet to plot our data.\nred_raster <- da_red %>% raster()\nred_raster\nThen plot the red band using plot function.\nplot(red_raster)" }, { - "objectID": "how-tos/access-data/access-local-python.html", - "href": "how-tos/access-data/access-local-python.html", - "title": "How do I access Cloud Data from my Local Machine in Python?", + "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html", + "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html", + "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", "section": "", - "text": "When you have found the data you want to use, you have two options. You can download the data to work locally, or access the data directly to work in the cloud. This second way of working is called “Direct Cloud Access” or simply, “Direct Access”.\nThis page covers downloading data from Earthdata Cloud to your local machine using the earthaccess library." + "text": "In this notebook, we will access monthly sea surface height from ECCO V4r4 (10.5067/ECG5D-SSH44). The data are provided as a time series of monthly netCDFs on a 0.5-degree latitude/longitude grid.\nWe will access a single netCDF file from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataset. This approach leverages S3 native protocols for efficient access to the data." }, { - "objectID": "how-tos/access-data/access-local-python.html#downloading-data", - "href": "how-tos/access-data/access-local-python.html#downloading-data", - "title": "How do I access Cloud Data from my Local Machine in Python?", + "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#summary", + "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#summary", + "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", "section": "", - "text": "When you have found the data you want to use, you have two options. You can download the data to work locally, or access the data directly to work in the cloud. This second way of working is called “Direct Cloud Access” or simply, “Direct Access”.\nThis page covers downloading data from Earthdata Cloud to your local machine using the earthaccess library." + "text": "In this notebook, we will access monthly sea surface height from ECCO V4r4 (10.5067/ECG5D-SSH44). The data are provided as a time series of monthly netCDFs on a 0.5-degree latitude/longitude grid.\nWe will access a single netCDF file from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataset. This approach leverages S3 native protocols for efficient access to the data." }, { - "objectID": "how-tos/access-data/access-local-python.html#earthaccess", - "href": "how-tos/access-data/access-local-python.html#earthaccess", - "title": "How do I access Cloud Data from my Local Machine in Python?", - "section": "earthaccess", - "text": "earthaccess\nWe can use the earthaccess python library to grab the file URLs and then access them with the xarray library.\n\n#Import packages\nimport earthaccess\nimport xarray as xr\n\n\n#Authentication with Earthdata Login\nauth = earthaccess.login(strategy=\"netrc\")\n\nYou're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 10/30/2023\nUsing .netrc file for EDL\n\n\n\n#Access land ice height from ATLAS/ICESat-2 V005 (10.5067/ATLAS/ATL06.005), searching for data over western Greenland coast over two weeks in July 2022. The data are provided as HDF5 granules (files) that span about 1/14th of an orbit.\n\nresults = earthaccess.search_data(short_name=\"ATL06\",\n version=\"005\",\n cloud_hosted=True,\n temporal = (\"2022-07-17\",\"2022-07-31\"),\n bounding_box = (-51.96423,68.10554,-48.71969,70.70529))\n\nGranules found: 5\n\n\n\nfiles = earthaccess.download(results, \"./local_folder\")\n\ndownloaded_files = []\nfor f in files: \n file = \"local_folder/\"+ f\n downloaded_files.append(file)\n\n Getting 5 granules, approx download size: 0.0 GB\nFile ATL06_20220718010029_03991603_005_01.h5 already downloaded\nFile ATL06_20220718134522_04071605_005_01.h5 already downloaded\nFile ATL06_20220722005209_04601603_005_01.h5 already downloaded\nFile ATL06_20220726004352_05211603_005_01.h5 already downloaded\nFile ATL06_20220722133704_04681605_005_01.h5 already downloaded\n\n\n\n\n\n\n\n\n\n\n\n['local_folder/ATL06_20220718010029_03991603_005_01.h5',\n 'local_folder/ATL06_20220718134522_04071605_005_01.h5',\n 'local_folder/ATL06_20220722005209_04601603_005_01.h5',\n 'local_folder/ATL06_20220722133704_04681605_005_01.h5',\n 'local_folder/ATL06_20220726004352_05211603_005_01.h5']\n\n\n\n#Use xarray to load the data as a multifile dataset for a single group in the HDF5 file, in this case land ice segments:\nds = xr.open_mfdataset(downloaded_files, group='/gt1l/land_ice_segments', engine='h5netcdf')\nds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (delta_time: 241711)\nCoordinates:\n * delta_time (delta_time) datetime64[ns] 2022-07-18T01:00:46.67...\n latitude (delta_time) float64 dask.array<chunksize=(78325,), meta=np.ndarray>\n longitude (delta_time) float64 dask.array<chunksize=(78325,), meta=np.ndarray>\nData variables:\n atl06_quality_summary (delta_time) int8 dask.array<chunksize=(78325,), meta=np.ndarray>\n h_li (delta_time) float32 dask.array<chunksize=(78325,), meta=np.ndarray>\n h_li_sigma (delta_time) float32 dask.array<chunksize=(78325,), meta=np.ndarray>\n segment_id (delta_time) float64 dask.array<chunksize=(78325,), meta=np.ndarray>\n sigma_geo_h (delta_time) float32 dask.array<chunksize=(78325,), meta=np.ndarray>\nAttributes:\n Description: The land_ice_height group contains the primary set of deriv...\n data_rate: Data within this group are sparse. Data values are provide...xarray.DatasetDimensions:delta_time: 241711Coordinates: (3)delta_time(delta_time)datetime64[ns]2022-07-18T01:00:46.678760592 .....contentType :referenceInformationdescription :Number of GPS seconds since the ATLAS SDP epoch. The ATLAS Standard Data Products (SDP) epoch offset is defined within /ancillary_data/atlas_sdp_gps_epoch as the number of GPS seconds between the GPS epoch (1980-01-06T00:00:00.000000Z UTC) and the ATLAS SDP epoch. By adding the offset contained within atlas_sdp_gps_epoch to delta time parameters, the time in gps_seconds relative to the GPS epoch can be computed.long_name :Elapsed GPS secondssource :section 4.4standard_name :timearray(['2022-07-18T01:00:46.678760592', '2022-07-18T01:00:46.681322640',\n '2022-07-18T01:00:46.684008720', ..., '2022-07-26T00:49:18.826637808',\n '2022-07-26T00:49:18.829449568', '2022-07-26T00:49:18.832263232'],\n dtype='datetime64[ns]')latitude(delta_time)float64dask.array<chunksize=(78325,), meta=np.ndarray>contentType :physicalMeasurementdescription :Latitude of segment center, WGS84, North=+,long_name :Latitudesource :section 3.10standard_name :latitudeunits :degrees_northvalid_max :90.0valid_min :-90.0\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n1.84 MiB\n611.91 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat64 numpy.ndarray\n\n\n\n\n\n\n\n\nlongitude\n\n\n(delta_time)\n\n\nfloat64\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nphysicalMeasurement\n\ndescription :\n\nLongitude of segment center, , WGS84, East=+\n\nlong_name :\n\nLongitude\n\nsource :\n\nsection 3.10\n\nstandard_name :\n\nlongitude\n\nunits :\n\ndegrees_east\n\nvalid_max :\n\n180.0\n\nvalid_min :\n\n-180.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n1.84 MiB\n611.91 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat64 numpy.ndarray\n\n\n\n\n\n\n\n\nData variables: (5)\n\n\n\n\n\natl06_quality_summary\n\n\n(delta_time)\n\n\nint8\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nqualityInformation\n\ndescription :\n\nThe ATL06_quality_summary parameter indicates the best-quality subset of all ATL06 data. A zero in this parameter implies that no data-quality tests have found a problem with the segment, a one implies that some potential problem has been found. Users who select only segments with zero values for this flag can be relatively certain of obtaining high-quality data, but will likely miss a significant fraction of usable data, particularly in cloudy, rough, or low-surface-reflectance conditions.\n\nflag_meanings :\n\nbest_quality potential_problem\n\nflag_values :\n\n[0 1]\n\nlong_name :\n\nATL06_Quality_Summary\n\nsource :\n\nsection 4.3\n\nunits :\n\n1\n\nvalid_max :\n\n1\n\nvalid_min :\n\n0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n236.05 kiB\n76.49 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nint8 numpy.ndarray\n\n\n\n\n\n\n\n\n\nh_li\n\n\n(delta_time)\n\n\nfloat32\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nphysicalMeasurement\n\ndescription :\n\nStandard land-ice segment height determined by land ice algorithm, corrected for first-photon bias, representing the median- based height of the selected PEs\n\nlong_name :\n\nLand Ice height\n\nsource :\n\nsection 4.4\n\nunits :\n\nmeters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.92 MiB\n305.96 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nh_li_sigma\n\n\n(delta_time)\n\n\nfloat32\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nqualityInformation\n\ndescription :\n\nPropagated error due to sampling error and FPB correction from the land ice algorithm\n\nlong_name :\n\nExpected RMS segment misfit\n\nsource :\n\nsection 4.4\n\nunits :\n\nmeters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.92 MiB\n305.96 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nsegment_id\n\n\n(delta_time)\n\n\nfloat64\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nreferenceInformation\n\ndescription :\n\nSegment number, counting from the equator. Equal to the segment_id for the second of the two 20m ATL03 segments included in the 40m ATL06 segment\n\nlong_name :\n\nReference Point, m\n\nsource :\n\nsection 3.1.2.1\n\nunits :\n\n1\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n1.84 MiB\n611.91 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat64 numpy.ndarray\n\n\n\n\n\n\n\n\n\nsigma_geo_h\n\n\n(delta_time)\n\n\nfloat32\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nqualityInformation\n\ndescription :\n\nTotal vertical geolocation error due to PPD and POD, including the effects of horizontal geolocation error on the segment vertical error.\n\nlong_name :\n\nVertical Geolocation Error\n\nsource :\n\nATBD Section 3.10\n\nunits :\n\nmeters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.92 MiB\n305.96 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nIndexes: (1)delta_timePandasIndexPandasIndex(DatetimeIndex(['2022-07-18 01:00:46.678760592',\n '2022-07-18 01:00:46.681322640',\n '2022-07-18 01:00:46.684008720',\n '2022-07-18 01:00:46.686753504',\n '2022-07-18 01:00:46.689526560',\n '2022-07-18 01:00:46.692315280',\n '2022-07-18 01:00:46.695049040',\n '2022-07-18 01:00:46.700724096',\n '2022-07-18 01:00:46.703545872',\n '2022-07-18 01:00:46.706366832',\n ...\n '2022-07-26 00:49:18.806914512',\n '2022-07-26 00:49:18.809737328',\n '2022-07-26 00:49:18.812559600',\n '2022-07-26 00:49:18.815380608',\n '2022-07-26 00:49:18.818200224',\n '2022-07-26 00:49:18.821015744',\n '2022-07-26 00:49:18.823827088',\n '2022-07-26 00:49:18.826637808',\n '2022-07-26 00:49:18.829449568',\n '2022-07-26 00:49:18.832263232'],\n dtype='datetime64[ns]', name='delta_time', length=241711, freq=None))Attributes: (2)Description :The land_ice_height group contains the primary set of derived ATL06 products. This includes geolocation, height, and standard error and quality measures for each segment. This group is sparse, meaning that parameters are provided only for pairs of segments for which at least one beam has a valid surface-height measurement.data_rate :Data within this group are sparse. Data values are provided only for those ICESat-2 20m segments where at least one beam has a valid land ice height measurement." + "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#requirements", + "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#requirements", + "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", + "section": "Requirements", + "text": "Requirements\n\n1. AWS instance running in us-west-2\nNASA Earthdata Cloud data in S3 can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region.\n\n\n2. Earthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\n3. netrc File\nYou will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. For additional information see: Authentication for NASA Earthdata." }, { - "objectID": "how-tos/plot.html", - "href": "how-tos/plot.html", - "title": "How do I plot data?", - "section": "", - "text": "Coming Soon!" + "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#learning-objectives", + "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#learning-objectives", + "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", + "section": "Learning Objectives", + "text": "Learning Objectives\n\nhow to retrieve temporary S3 credentials for in-region direct S3 bucket access\nhow to perform in-region direct access of ECCO_L4_SSH_05DEG_MONTHLY_V4R4 data in S3\nhow to plot the data" }, { - "objectID": "how-tos/plot.html#how-do-i-plot-a-netcdf-variable", - "href": "how-tos/plot.html#how-do-i-plot-a-netcdf-variable", - "title": "How do I plot data?", - "section": "How do I plot a NetCDF variable?", - "text": "How do I plot a NetCDF variable?" + "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#import-packages", + "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#import-packages", + "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", + "section": "Import Packages", + "text": "Import Packages\n\n%matplotlib inline\nimport matplotlib.pyplot as plt\nimport os\nimport requests\nimport s3fs\nfrom osgeo import gdal\nimport xarray as xr\nimport hvplot.xarray\nimport holoviews as hv" }, { - "objectID": "how-tos/subset.html", - "href": "how-tos/subset.html", - "title": "How do I subset data granules?", - "section": "", - "text": "PythonRMATLABCommand Line\n\n\nInstall the harmony-py package:\n# Install harmony-py\npip install -U harmony-py\nImport packages:\nimport datetime as dt\n\nfrom harmony import BBox, Client, Collection, Request, LinkType\n\nimport s3fs\nimport xarray as xr\n\n\nWe will authenticate the following Harmony request using a netrc file. See the appendix for more information on Earthdata Login and netrc setup. This basic line below to create a Harmony Client assumes that we have a .netrc available.\nharmony_client = Client()\n\n\n\nWe are interested in the GHRSST Level 4 MUR Global Foundation Sea Surface Temperature Analysis dataset. We are subsetting over the Pacific Ocean to the west of Mexico during 1:00 - 2:00 on 10 March 2021. The dataset is organized into daily files, so while we are specifying a single hour in our request, this will return that full day’s worth of data.\ndataset_short_name = 'MUR-JPL-L4-GLOB-v4.1'\n\nrequest = Request(\n collection=Collection(id=dataset_short_name),\n spatial=BBox(-125.469,15.820,-99.453,35.859),\n temporal={\n 'start': dt.datetime(2021, 3, 10, 1),\n 'stop': dt.datetime(2021, 3, 10, 2)\n }\n)\n\njob_id = harmony_client.submit(request)\n\nharmony_client.wait_for_processing(job_id)\n\n\n\nHarmony data outputs can be accessed within the cloud using the s3 URLs and AWS credentials provided in the Harmony job response. Using aws_credentials we can retrieve the credentials needed to access the Harmony s3 staging bucket and its contents. We then use the AWS s3fs package to create a file system that can then be read by xarray.\nresults = harmony_client.result_urls(job_id, link_type=LinkType.s3)\nurls = list(results)\nurl = urls[0]\n\ncreds = harmony_client.aws_credentials()\n\ns3_fs = s3fs.S3FileSystem(\n key=creds['aws_access_key_id'],\n secret=creds['aws_secret_access_key'],\n token=creds['aws_session_token'],\n client_kwargs={'region_name':'us-west-2'},\n)\n\nf = s3_fs.open(url, mode='rb')\nds = xr.open_dataset(f)\nds\n\n\n\nUse the xarray built in plotting function to create a simple plot along the x and y dimensions of the dataset:\nds.analysed_sst.plot() ;\n\n\n\nR code coming soon!\n# Coming soon!\n\n\nThis example assumes that you have experimented with and understood the tutorial MATLAB Access Single NASA EarthData L2 NetCDF presented in this Cookbook.\nTo subset a dataset, either read it over its full extent and subset it using MATLAB indexing (a posteriori, client-side), or use the start, count, and stride parameters of function h5read to read only specific regions/samples (hyperslabs) of it (server side). The code snippets that follow present both approaches, for accessing a dataset of sea surface temperature.\nFollow the authentication procedure presented in the tutorial:\naddpath(\"<PATH_TO_FUNCTION_LOADAWSCREDENTIALSENPOINT>\") ;\ndaacCredentialsEndpoint = \"https://archive.podaac.earthdata.nasa.gov/s3credentials\";\nloadAWSCredentials(daacCredentialsEndpoint);\nDefine relevant file and dataset:\nfilePath = \"s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20100619062008-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\";\ndatasetName = \"sea_surface_temperature_4um\";\ndatasetPath = \"/\" + datasetName;\nRead full dataset, longitudes, and latitudes. Replace fill values by NaN (so they are not displayed by MALTAB), and build contour plot:\nfill_value = h5readatt(filePath,datasetPath,\"_FillValue\");\nname = h5readatt(filePath,datasetPath,\"long_name\");\ndata = h5read(filePath,datasetPath);\ndata(data == fill_value) = NaN;\nlat = h5read(filePath,\"/lat\");\nlon = h5read(filePath,\"/lon\");\ncontour(lon,lat,data);\ntitle(name);\nSubset/sub-sample full dataset with a step of 10 (a posteriori) using MATLAB indexing:\nstep = 10;\ndataSize = size(data);\nrowIx = 1 : step : dataSize(1);\ncolIx = 1 : step : dataSize(2);\ncontour(lon(rowIx,colIx),lat(rowIx,colIx),data(rowIx,colIx));\ntitle(name + \" (sub-sampled through MATLAB indexing)\");\nUse the start, count, and stride parameters of function h5read to read only specific regions/samples (hyperslabs) of it (server side). These parameters must be defined as vectors whose size matches the number of dimensions of the dataset. Here, we want to read data starting at index 1, with a stride/step of 10, and cover the full extent of the dataset (count = Inf). Based on our experiments above, we assume that the dataset is a 2D array:\ndataStride10 = h5read(filePath,datasetPath,[1,1],[Inf,Inf],[step,step]);\nRunning this leads to the following error:\nError using h5readc\nThe START, COUNT, and STRIDE parameters must be double precision, positive, and have length equal to the rank of the dataset.\nThis error message is too succint to be helpful. Try to get more information about the dataset using the h5disp, which is cloud-enabled unlike ncdisp:\nh5disp(filePath,datasetPath)\n\nOUTPUT:\nHDF5 20100619062008-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc \nDataset 'sea_surface_temperature_4um' \n Size: 1354x2030x1 <--- Shows a 3rd (singleton) dimension\n MaxSize: 1354x2030x1 \n Datatype: H5T_STD_I16LE (int16)\n ChunkSize: 677x1015x1\n Filters: deflate(5)\n FillValue: -32767\n Attributes:\n 'long_name': 'sea surface temperature'\n 'units': 'kelvin'\n '_FillValue': -32767 \n 'valid_min': -1000 \n 'valid_max': 10000 \n 'comment': 'sea surface temperature from mid-IR (4 um) channels; non L2P core field'\n 'scale_factor': 0.005000 \n 'add_offset': 273.149994 \n 'coordinates': 'lon lat'\n 'coverage_content_type': 'physicalMeasurement'\n 'DIMENSION_LIST': H5T_VLEN\nThis output shows a 3rd (singleton) dimension, but it still does not explain what it is.\nOne workaround is to display more of the content of the file through h5disp and/or h5info and try to get relevant information about dimensions. The following shows a simpler approach, however: create a local copy of the file and analyze it using ncdisp. It is possible because MATLAB IO functions, and in particular copyfile, are cloud-enabled:\ncopyfile(filePath,\"test.nc\");\nncdisp(\"test.nc\",datasetName);\n\nOUTPUT:\nSource:\n .../test.nc\nFormat:\n netcdf4\nDimensions:\n ni = 1354\n nj = 2030\n time = 1\nVariables:\n sea_surface_temperature_4um\n Size: 1354x2030x1\n Dimensions: ni,nj,time\n Datatype: int16\n Attributes:\n ...\nThis makes it clear that there is a 3rd dimension that is time, with a size of 1.\nSubset the dataset using h5read and vectors of 3 elements that define start, count, and stride for the 3 dimensions:\ndataStride10 = h5read(filePath,datasetPath,[1,1,1],[Inf,Inf,Inf],[step,step,1]);\ndataStride10(dataStride10 == fill_value) = NaN;\nlatStride10 = h5read(filePath,\"/lat\",[1,1],[Inf,Inf],[step,step]);\nlonStride10 = h5read(filePath,\"/lon\",[1,1],[Inf,Inf],[step,step]);\ncontour(lonStride10,latStride10,dataStride10);\ntitle(name+ \" (sub-sampled through H5READ stride parameter)\");\nWe can target a specific region using the same two approaches. The following shows how to create an array of logicals that “flags” a region of interest, how to identify relevant rows and columns of the data, latitudes, and longitudes arrays, and how to use logical indexing for plotting the data:\nlatBounds = [31,33.5];\nlonBounds = [-64,-57.5];\nisRegion = lat>=latBounds(1) & lat<=latBounds(2) & lon>=lonBounds(1) & lon<=lonBounds(2);\nisRow = any(isRegion,2);\nisCol = any(isRegion,1);\ncontour(lon(isRow,isCol),lat(isRow,isCol),data(isRow,isCol));\nxlim(lonBounds);\nylim(latBounds);\ntitle(name+ \" (selected region)\");\n\n\nWith wget and curl:\n# Coming soon!" + "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#get-temporary-aws-credentials", + "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#get-temporary-aws-credentials", + "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", + "section": "Get Temporary AWS Credentials", + "text": "Get Temporary AWS Credentials\nDirect S3 access is achieved by passing NASA supplied temporary credentials to AWS so we can interact with S3 objects from applicable Earthdata Cloud buckets. For now, each NASA DAAC has different AWS credentials endpoints. Below are some of the credential endpoints to various DAACs:\n\ns3_cred_endpoint = {\n 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n 'gesdisc': 'https://data.gesdisc.earthdata.nasa.gov/s3credentials',\n 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n 'ornldaac': 'https://data.ornldaac.earthdata.nasa.gov/s3credentials',\n 'ghrcdaac': 'https://data.ghrc.earthdata.nasa.gov/s3credentials'\n}\n\nCreate a function to make a request to an endpoint for temporary credentials. Remember, each DAAC has their own endpoint and credentials are not usable for cloud data from other DAACs.\n\ndef get_temp_creds(provider):\n return requests.get(s3_cred_endpoint[provider]).json()\n\n\ntemp_creds_req = get_temp_creds('podaac')\n#temp_creds_req" }, { - "objectID": "how-tos/subset.html#how-do-i-subset-a-data-granule-using-harmony", - "href": "how-tos/subset.html#how-do-i-subset-a-data-granule-using-harmony", - "title": "How do I subset data granules?", - "section": "", - "text": "PythonRMATLABCommand Line\n\n\nInstall the harmony-py package:\n# Install harmony-py\npip install -U harmony-py\nImport packages:\nimport datetime as dt\n\nfrom harmony import BBox, Client, Collection, Request, LinkType\n\nimport s3fs\nimport xarray as xr\n\n\nWe will authenticate the following Harmony request using a netrc file. See the appendix for more information on Earthdata Login and netrc setup. This basic line below to create a Harmony Client assumes that we have a .netrc available.\nharmony_client = Client()\n\n\n\nWe are interested in the GHRSST Level 4 MUR Global Foundation Sea Surface Temperature Analysis dataset. We are subsetting over the Pacific Ocean to the west of Mexico during 1:00 - 2:00 on 10 March 2021. The dataset is organized into daily files, so while we are specifying a single hour in our request, this will return that full day’s worth of data.\ndataset_short_name = 'MUR-JPL-L4-GLOB-v4.1'\n\nrequest = Request(\n collection=Collection(id=dataset_short_name),\n spatial=BBox(-125.469,15.820,-99.453,35.859),\n temporal={\n 'start': dt.datetime(2021, 3, 10, 1),\n 'stop': dt.datetime(2021, 3, 10, 2)\n }\n)\n\njob_id = harmony_client.submit(request)\n\nharmony_client.wait_for_processing(job_id)\n\n\n\nHarmony data outputs can be accessed within the cloud using the s3 URLs and AWS credentials provided in the Harmony job response. Using aws_credentials we can retrieve the credentials needed to access the Harmony s3 staging bucket and its contents. We then use the AWS s3fs package to create a file system that can then be read by xarray.\nresults = harmony_client.result_urls(job_id, link_type=LinkType.s3)\nurls = list(results)\nurl = urls[0]\n\ncreds = harmony_client.aws_credentials()\n\ns3_fs = s3fs.S3FileSystem(\n key=creds['aws_access_key_id'],\n secret=creds['aws_secret_access_key'],\n token=creds['aws_session_token'],\n client_kwargs={'region_name':'us-west-2'},\n)\n\nf = s3_fs.open(url, mode='rb')\nds = xr.open_dataset(f)\nds\n\n\n\nUse the xarray built in plotting function to create a simple plot along the x and y dimensions of the dataset:\nds.analysed_sst.plot() ;\n\n\n\nR code coming soon!\n# Coming soon!\n\n\nThis example assumes that you have experimented with and understood the tutorial MATLAB Access Single NASA EarthData L2 NetCDF presented in this Cookbook.\nTo subset a dataset, either read it over its full extent and subset it using MATLAB indexing (a posteriori, client-side), or use the start, count, and stride parameters of function h5read to read only specific regions/samples (hyperslabs) of it (server side). The code snippets that follow present both approaches, for accessing a dataset of sea surface temperature.\nFollow the authentication procedure presented in the tutorial:\naddpath(\"<PATH_TO_FUNCTION_LOADAWSCREDENTIALSENPOINT>\") ;\ndaacCredentialsEndpoint = \"https://archive.podaac.earthdata.nasa.gov/s3credentials\";\nloadAWSCredentials(daacCredentialsEndpoint);\nDefine relevant file and dataset:\nfilePath = \"s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20100619062008-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\";\ndatasetName = \"sea_surface_temperature_4um\";\ndatasetPath = \"/\" + datasetName;\nRead full dataset, longitudes, and latitudes. Replace fill values by NaN (so they are not displayed by MALTAB), and build contour plot:\nfill_value = h5readatt(filePath,datasetPath,\"_FillValue\");\nname = h5readatt(filePath,datasetPath,\"long_name\");\ndata = h5read(filePath,datasetPath);\ndata(data == fill_value) = NaN;\nlat = h5read(filePath,\"/lat\");\nlon = h5read(filePath,\"/lon\");\ncontour(lon,lat,data);\ntitle(name);\nSubset/sub-sample full dataset with a step of 10 (a posteriori) using MATLAB indexing:\nstep = 10;\ndataSize = size(data);\nrowIx = 1 : step : dataSize(1);\ncolIx = 1 : step : dataSize(2);\ncontour(lon(rowIx,colIx),lat(rowIx,colIx),data(rowIx,colIx));\ntitle(name + \" (sub-sampled through MATLAB indexing)\");\nUse the start, count, and stride parameters of function h5read to read only specific regions/samples (hyperslabs) of it (server side). These parameters must be defined as vectors whose size matches the number of dimensions of the dataset. Here, we want to read data starting at index 1, with a stride/step of 10, and cover the full extent of the dataset (count = Inf). Based on our experiments above, we assume that the dataset is a 2D array:\ndataStride10 = h5read(filePath,datasetPath,[1,1],[Inf,Inf],[step,step]);\nRunning this leads to the following error:\nError using h5readc\nThe START, COUNT, and STRIDE parameters must be double precision, positive, and have length equal to the rank of the dataset.\nThis error message is too succint to be helpful. Try to get more information about the dataset using the h5disp, which is cloud-enabled unlike ncdisp:\nh5disp(filePath,datasetPath)\n\nOUTPUT:\nHDF5 20100619062008-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc \nDataset 'sea_surface_temperature_4um' \n Size: 1354x2030x1 <--- Shows a 3rd (singleton) dimension\n MaxSize: 1354x2030x1 \n Datatype: H5T_STD_I16LE (int16)\n ChunkSize: 677x1015x1\n Filters: deflate(5)\n FillValue: -32767\n Attributes:\n 'long_name': 'sea surface temperature'\n 'units': 'kelvin'\n '_FillValue': -32767 \n 'valid_min': -1000 \n 'valid_max': 10000 \n 'comment': 'sea surface temperature from mid-IR (4 um) channels; non L2P core field'\n 'scale_factor': 0.005000 \n 'add_offset': 273.149994 \n 'coordinates': 'lon lat'\n 'coverage_content_type': 'physicalMeasurement'\n 'DIMENSION_LIST': H5T_VLEN\nThis output shows a 3rd (singleton) dimension, but it still does not explain what it is.\nOne workaround is to display more of the content of the file through h5disp and/or h5info and try to get relevant information about dimensions. The following shows a simpler approach, however: create a local copy of the file and analyze it using ncdisp. It is possible because MATLAB IO functions, and in particular copyfile, are cloud-enabled:\ncopyfile(filePath,\"test.nc\");\nncdisp(\"test.nc\",datasetName);\n\nOUTPUT:\nSource:\n .../test.nc\nFormat:\n netcdf4\nDimensions:\n ni = 1354\n nj = 2030\n time = 1\nVariables:\n sea_surface_temperature_4um\n Size: 1354x2030x1\n Dimensions: ni,nj,time\n Datatype: int16\n Attributes:\n ...\nThis makes it clear that there is a 3rd dimension that is time, with a size of 1.\nSubset the dataset using h5read and vectors of 3 elements that define start, count, and stride for the 3 dimensions:\ndataStride10 = h5read(filePath,datasetPath,[1,1,1],[Inf,Inf,Inf],[step,step,1]);\ndataStride10(dataStride10 == fill_value) = NaN;\nlatStride10 = h5read(filePath,\"/lat\",[1,1],[Inf,Inf],[step,step]);\nlonStride10 = h5read(filePath,\"/lon\",[1,1],[Inf,Inf],[step,step]);\ncontour(lonStride10,latStride10,dataStride10);\ntitle(name+ \" (sub-sampled through H5READ stride parameter)\");\nWe can target a specific region using the same two approaches. The following shows how to create an array of logicals that “flags” a region of interest, how to identify relevant rows and columns of the data, latitudes, and longitudes arrays, and how to use logical indexing for plotting the data:\nlatBounds = [31,33.5];\nlonBounds = [-64,-57.5];\nisRegion = lat>=latBounds(1) & lat<=latBounds(2) & lon>=lonBounds(1) & lon<=lonBounds(2);\nisRow = any(isRegion,2);\nisCol = any(isRegion,1);\ncontour(lon(isRow,isCol),lat(isRow,isCol),data(isRow,isCol));\nxlim(lonBounds);\nylim(latBounds);\ntitle(name+ \" (selected region)\");\n\n\nWith wget and curl:\n# Coming soon!" + "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#set-up-an-s3fs-session-for-direct-access", + "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#set-up-an-s3fs-session-for-direct-access", + "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", + "section": "Set up an s3fs session for Direct Access", + "text": "Set up an s3fs session for Direct Access\ns3fs sessions are used for authenticated access to s3 bucket and allows for typical file-system style operations. Below we create session by passing in the temporary credentials we recieved from our temporary credentials endpoint.\n\nfs_s3 = s3fs.S3FileSystem(anon=False, \n key=temp_creds_req['accessKeyId'], \n secret=temp_creds_req['secretAccessKey'], \n token=temp_creds_req['sessionToken'])\n\nIn this example we’re interested in the ECCO data collection from NASA’s PO.DAAC in Earthdata Cloud. Below we specify the s3 URL to the data asset in Earthdata Cloud. This URL can be found via Earthdata Search or programmatically through the CMR and CMR-STAC APIs.\n\ns3_url = 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-01_ECCO_V4r4_latlon_0p50deg.nc'" }, { - "objectID": "how-tos/subset.html#how-do-i-subset-an-opendap-granule-in-the-cloud", - "href": "how-tos/subset.html#how-do-i-subset-an-opendap-granule-in-the-cloud", - "title": "How do I subset data granules?", - "section": "How do I subset an OPeNDAP granule in the cloud?", - "text": "How do I subset an OPeNDAP granule in the cloud?" + "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#direct-in-region-access", + "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#direct-in-region-access", + "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", + "section": "Direct In-region Access", + "text": "Direct In-region Access\nOpen with the netCDF file using the s3fs package, then load the cloud asset into an xarray dataset.\n\ns3_file_obj = fs_s3.open(s3_url, mode='rb')\n\n\nssh_ds = xr.open_dataset(s3_file_obj, engine='h5netcdf')\nssh_ds\n\nGet the SSH variable as an xarray dataarray\n\nssh_da = ssh_ds.SSH\nssh_da\n\nPlot the SSH dataarray for time 2015-01-16T12:00:00 using hvplot.\n\nssh_da.hvplot.image(x='longitude', y='latitude', cmap='Spectral_r', geo=True, tiles='ESRI', global_extent=True)" }, { - "objectID": "how-tos/subset.html#how-do-i-subset-a-data-granule-using-xarray", - "href": "how-tos/subset.html#how-do-i-subset-a-data-granule-using-xarray", - "title": "How do I subset data granules?", - "section": "How do I subset a data granule using xarray?", - "text": "How do I subset a data granule using xarray?" + "objectID": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#resources", + "href": "appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html#resources", + "title": "Accessing a NetCDF4/HDF5 File - S3 Direct Access", + "section": "Resources", + "text": "Resources\nDirect access to ECCO data in S3 (from us-west-2)\nData_Access__Direct_S3_Access__PODAAC_ECCO_SSH using CMR-STAC API to retrieve S3 links" }, { - "objectID": "contributing/environment.html", - "href": "contributing/environment.html", - "title": "JupyterHub environment", + "objectID": "leading-workshops/index.html", + "href": "leading-workshops/index.html", + "title": "Leading Workshops", + "section": "", + "text": "This section, under development, will include information for workshop prep, setup, and hosting." + }, + { + "objectID": "leading-workshops/index.html#using-the-openscapes-2i2c-hub-in-a-workshop", + "href": "leading-workshops/index.html#using-the-openscapes-2i2c-hub-in-a-workshop", + "title": "Leading Workshops", + "section": "Using the Openscapes 2i2c Hub in a workshop", + "text": "Using the Openscapes 2i2c Hub in a workshop\n\nCheck with Luis that the Hub image has the packages you need\nReach out to 2i2c a month in advance via email support at 2i2c.freshdesk.com (example below) to tell them about the workshop date, start and end times, # of participants, anticipated level of resources to be used.\nAdd participants to the 2i2c Hub via a GitHub Team\n\n\n\n\n\n\n\nExample email to 2i2c for a workshop\n\n\n\n\n\nHello,\nI wanted to reach out to the support desk to let you know that Openscapes will have a workshop using Python 2i2c instances for about 50 attendees.\nTitle: Data Access Workshop for NASA’s SWOT Satellite\nDate: February 13, 2024\nDuration/Time: half day, 9:00 am-12:30 pm HST (Honolulu, HI).\nExpected Usage: 3.7 GB per person (~50 people)\nThank you!\nCassie\n\n\n\n\nHow to Add Folks to the 2i2c Hub" + }, + { + "objectID": "contributing/index.html", + "href": "contributing/index.html", + "title": "Contributing", "section": "", - "text": "Our JupyterHub is managed by 2i2c; see background from the 2021 Cloud Hackathon [JupyterHub intro]](https://nasa-openscapes.github.io/2021-Cloud-Hackathon/clinic/jupyterhub.html).\nWe create the environment with software packages for everyone to use - this includes python packages and quarto. Our environment is called corn, developed by NSIDC software engineer Luis Lopez. Here are some notes about corn; please refer to the GitHub README for details about installation.\nWhen looking at environment.yml: - Main dependency is python. Python 3.9 has some stability “pinned” the version - Next thing: pangeo-notebook. This is not pinned to a version, whatever their latest is best. This is what makes environment run in 2i2c - The rest are optional, they’ve been added as different groups want them - It runs on linux-64 and mac-OS - side note: ~ means “don’t update past breaking versions”.\nNow getting to a point where they are conflicting and can get into problems with deprecation and failing the environment. Next strategy: trim down “core” corn environment. Then have a process & writeup for teams to copy that environment and add their additional packages, and make sure it’s the one running in 2i2c. Point to existing resources as much as possible (i.e. Turing Way, Pangeo, Pythia); link to them from here and provide our-case-specific details here." + "text": "The NASA Openscapes Mentor Community is collaborating to create this Cookbook, with an eye towards how others could collaborate with us in the future.\nThe Openscapes Approach Guide describes how we work, including creating space and place, having a growth mindset, psychological safety, and continual learning, practice, and iteration. It also has our code of conduct and an overview of some of the workflows and tooling we use.\nWe believe open practices can accelerate data-driven solutions and increase diversity, equity, inclusion, and belonging in science. These are critical to uncovering enduring science-based solutions faster, as well as for the well-being and resilience of research communities. We work towards kinder science.\nOur style of working is greatly influenced by:\n\nThe Turing Way Community Handbook\nThe Carpentries Curriculum Development Handbook\nThe Documentation System - by Daniele Procido/Divio; see image below!\nRStudio Cheatsheets\neScience Hackweeks\n\nThe Earthdata Cloud Cookbook is made with Quarto with our 2i2c JupyterHub. Learn more in the Setup and Workflow sections.\n\n\n\nFigure from documentation.divio.com: There is a secret that needs to be understood in order to write good software documentation: there isn't one thing called documentation, there are four. They are: tutorials, how-to guides, technical reference and explanation. They represent four different purposes or functions, and require four different approaches to their creation." }, { - "objectID": "contributing/environment.html#introduction", - "href": "contributing/environment.html#introduction", - "title": "JupyterHub environment", + "objectID": "contributing/index.html#how-we-work", + "href": "contributing/index.html#how-we-work", + "title": "Contributing", "section": "", - "text": "Our JupyterHub is managed by 2i2c; see background from the 2021 Cloud Hackathon [JupyterHub intro]](https://nasa-openscapes.github.io/2021-Cloud-Hackathon/clinic/jupyterhub.html).\nWe create the environment with software packages for everyone to use - this includes python packages and quarto. Our environment is called corn, developed by NSIDC software engineer Luis Lopez. Here are some notes about corn; please refer to the GitHub README for details about installation.\nWhen looking at environment.yml: - Main dependency is python. Python 3.9 has some stability “pinned” the version - Next thing: pangeo-notebook. This is not pinned to a version, whatever their latest is best. This is what makes environment run in 2i2c - The rest are optional, they’ve been added as different groups want them - It runs on linux-64 and mac-OS - side note: ~ means “don’t update past breaking versions”.\nNow getting to a point where they are conflicting and can get into problems with deprecation and failing the environment. Next strategy: trim down “core” corn environment. Then have a process & writeup for teams to copy that environment and add their additional packages, and make sure it’s the one running in 2i2c. Point to existing resources as much as possible (i.e. Turing Way, Pangeo, Pythia); link to them from here and provide our-case-specific details here." + "text": "The NASA Openscapes Mentor Community is collaborating to create this Cookbook, with an eye towards how others could collaborate with us in the future.\nThe Openscapes Approach Guide describes how we work, including creating space and place, having a growth mindset, psychological safety, and continual learning, practice, and iteration. It also has our code of conduct and an overview of some of the workflows and tooling we use.\nWe believe open practices can accelerate data-driven solutions and increase diversity, equity, inclusion, and belonging in science. These are critical to uncovering enduring science-based solutions faster, as well as for the well-being and resilience of research communities. We work towards kinder science.\nOur style of working is greatly influenced by:\n\nThe Turing Way Community Handbook\nThe Carpentries Curriculum Development Handbook\nThe Documentation System - by Daniele Procido/Divio; see image below!\nRStudio Cheatsheets\neScience Hackweeks\n\nThe Earthdata Cloud Cookbook is made with Quarto with our 2i2c JupyterHub. Learn more in the Setup and Workflow sections.\n\n\n\nFigure from documentation.divio.com: There is a secret that needs to be understood in order to write good software documentation: there isn't one thing called documentation, there are four. They are: tutorials, how-to guides, technical reference and explanation. They represent four different purposes or functions, and require four different approaches to their creation." }, { - "objectID": "contributing/environment.html#more-background", - "href": "contributing/environment.html#more-background", - "title": "JupyterHub environment", - "section": "More background", - "text": "More background\nSome articles about Python environments with Conda that are interesting and related to our 2i2c workflow:\n\nhttps://www.anaconda.com/blog/understanding-and-improving-condas-performance\nhttps://medium.com/@marius.v.niekerk/conda-metachannel-f962241c9437\nhttps://wolfv.medium.com/making-conda-fast-again-4da4debfb3b7\nhttps://pythonspeed.com/articles/conda-dependency-management/\n\nWe are doing what article the fourth article describes. We are using conda-lock with versioned dependencies. Since corn was designed to run in 2i2c I never bothered to test it for windows or OSx but I think we should offer that possibility. I added OSX and will need some volunteers to test it on Windows." + "objectID": "contributing/community.html", + "href": "contributing/community.html", + "title": "Community Resources", + "section": "", + "text": "Running the NASA Cloud Workshop notebooks with mybinder.org - by Eli Holmes, 2021 Cloud Hackathon Participant who then set up working in Binder" }, { "objectID": "contributing/setup.html", @@ -2352,102 +2233,67 @@ "text": "Build Cookbook!\nWe use Quarto to build (“preview”) our Cookbook and publish it via GitHub. In the Terminal, your current directory should already be the cookbook (cd earthdata-cloud-cookbook). Type the following, which will open the Cookbook as a new tab in your browser:\nquarto preview\nBackground: Quarto is a new open-source scientific and technical publishing system. It makes collaborating to create technical documentation streamlined because we work in plain text documents that can have executable code (Python, R) and are rendered using Jupyter and Knitr engines. Explore Quarto and use it hands-on in our separate Quarto Website Tutorial. Quarto is already installed in our JupyterHub.\nNow you’re all set to contribute to the Cookbook! Read about how in the next chapter." }, { - "objectID": "contributing/onboarding.html", - "href": "contributing/onboarding.html", - "title": "Onboarding Mentors", + "objectID": "contributing/quarto-website-tutorial.html", + "href": "contributing/quarto-website-tutorial.html", + "title": "Quarto Tutorial", "section": "", - "text": "We expect many contributors are folks who are part of the NASA Openscapes Mentor Community, involved in user support services at the NASA EOSDIS Distributed Active Archive Centers (DAACs). This page is onboarding information for these Mentors. Further chapters describe how to onboard to all the tech and workflows that everyone can use to contribute to the Cookbook." + "text": "We’ve created a tutorial on how to copy (fork or download) an existing Quarto website like this one and adapt it for your own uses. Please see: https://openscapes.github.io/quarto-website-tutorial/." }, { - "objectID": "contributing/onboarding.html#welcome-to-the-mentor-community", - "href": "contributing/onboarding.html#welcome-to-the-mentor-community", - "title": "Onboarding Mentors", - "section": "Welcome to the mentor community!", - "text": "Welcome to the mentor community!\nAs we onboard more folks to grow our cross-DAAC NASA Mentors community, we will be documenting the onboarding process here. We expect Mentors to have different experiences, expertise, technical skills, and job responsibilities. Goals of onboarding will be:\n\nget to know each other, strengthen relationships across DAACs and NASA\nshare previous work by NASA Mentors\nidentify common needs and priorities\ndevelop common approaches and solutions\nstrengthen technical skills\nstrengthen teaching skills\nbuild habits and confidence contributing to common materials\nmeet Champions and other researchers to better design how we support them\n\nWe meet remotely, primarily through Mentor Calls (focus: learning and planning) and Coworking Calls (focus: screensharing and doing). We teach together through events like Hackathons, Workshops, and Openscapes Champions Cohorts." + "objectID": "contributing/troubleshooting.html", + "href": "contributing/troubleshooting.html", + "title": "Troubleshooting", + "section": "", + "text": "Starting advice from https://nasa-openscapes.github.io/2021-Cloud-Hackathon/logistics/github-workflows.html#git-update-revert-etc" }, { - "objectID": "contributing/onboarding.html#mentor-schedule", - "href": "contributing/onboarding.html#mentor-schedule", - "title": "Onboarding Mentors", - "section": "Mentor schedule", - "text": "Mentor schedule\nSee our schedule of events at https://nasa-openscapes.github.io/mentors" + "objectID": "contributing/troubleshooting.html#jupyterhub", + "href": "contributing/troubleshooting.html#jupyterhub", + "title": "Troubleshooting", + "section": "JupyterHub", + "text": "JupyterHub\nNot a git repository - in your terminal if you see the following, you likely need to cd change directory into your GitHub folder.\nfatal: not a git repository (or any parent up to mount point /home)\nStopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set)." }, { - "objectID": "contributing/onboarding.html#tooling-checklist", - "href": "contributing/onboarding.html#tooling-checklist", - "title": "Onboarding Mentors", - "section": "Tooling checklist", - "text": "Tooling checklist\nThe tooling we use to collaborate include the following. We don’t expect everyone to have the same level of skills across all software. We will hold skill-building clinics to help strengthen familiarity and confidence, with the goal that everyone is able to contribute through overlapping and interoperable skills.\nWhen you join, expect to get access to the following. We’ll track this in the OpenscapesParticipantsList [ nasa-mentors ] spreadsheet, which is in the Google Drive Folder. Note that to add you to GitHub and the 2i2c JupyterHub, we will need you to share your GitHub username see setup instructions.\n\n✅ Google Group nasa-openscapes-mentors(We use this list primarily for calendar invites and occasional emails to the whole group)\n✅ Google Calendar invitations (for Cohort Calls and Coworking)\n✅ Google Drive (for shared notes and presentations; see below)\n✅ Slack (for connecting to NASA Mentors and broader Openscapes community; see below)\n✅ GitHub (for sharing code and documentation, project management and code review; see Contibuting Setup). We will ask for your GitHub username and add you to the NASA-Openscapes Organization.\n✅ 2i2c JupyterHub (for Python, R, and Matlab; see Contibuting Setup). We will add you with your GitHub username.\n✅ Carpentries Instructor Training (for a common pedagogy for teaching learners code- and data- intensive science).\n\n\nOpenscapes Slack\nChannels to join:\n\nnasa-mentors (private)\n2023-nasa-champions (private)\n#cloud-infrastructure\n\nHere is deeper documentation about Openscapes Slack.\n\n\nMentors Google Drive\nWe have a shared Google Drive folder: NASA-OpenscapesMentors [ nasa-mentors ]. We’ll add you with edit privileges. There is a README file within it that orients you to the folders within. The convention is that within the [ brackets ] is who has permission within the folder.\n\n\nMentors GitHub Organization\nWe’ll add you to our NASA-Openscapes GitHub Organization.\n\n\nCarpentries Instructor Training\nAs a professional development opportunity for Mentors, we partner with The Carpentries, a nonprofit that teaches introductory coding skills around the world. Their remote Instructor Training is not coding-specific, but it is a pedagogical approach to teaching technical topics. If you would like to sign up, we have an access code that you can use to select the date-times via the Instructor Training event calendar." + "objectID": "contributing/troubleshooting.html#git-flight-rules", + "href": "contributing/troubleshooting.html#git-flight-rules", + "title": "Troubleshooting", + "section": "Git Flight Rules", + "text": "Git Flight Rules\n“A guide for astronauts (now, programmers using Git) about what to do when things go wrong.” This is quite a comprehensive and well-maintained list of git commands, using the command line.\nhttps://github.com/k88hudson/git-flight-rules" }, { - "objectID": "contributing/onboarding.html#background", - "href": "contributing/onboarding.html#background", - "title": "Onboarding Mentors", - "section": "Background", - "text": "Background\nNASA Openscapes is a cross-DAAC mentor community creating & teaching resources to help researchers migrate workflows to the cloud. We meet and learn together, working with alignment on common needs for researchers using NASA Earthdata. Mentors receive Carpentries Instructor training and Openscapes mentorship to help you refine teaching strategies for Cloud, while collaborating on cloud tutorials to reuse rather than reinventing separately.\nTogether, we will teach and support in opportunities of different types: Hackathons, Workshops, and the Openscapes Champions program. We expect a range of technical ability and teaching experience. Through these growth opportunities within a broader community engagement approach, you will also gain visibility for your work as well as for your DAACs. The expected time commitment is 3-4 days/month per year. This accounts for a minimum of 4 hours/month of synchronous calls with the rest of the time being self-directed towards improving support approaches for your specific DAAC.\n\nResources\nTo get a broader sense of the NASA Openscapes Community’s work, please review these recent resources. View the full list of talks and workshops at https://nasa-openscapes.github.io.\n\nNASA Openscapes: Supporting NASA EarthData Users Transition to the Cloud - November 2 2023. Erin Robinson, GHRC User Working Group\nNASA Openscapes Mentors’ Retreat 2023: Summary and Moving Toward Sustainability - September 18, 2023. Catalina Taglialatela and the NASA Openscapes Mentors, NASA HQ Meeting\nWorking with NASA Earthdata in the Cloud- January 24, 2023 - Amy Steiker, Bri Lind, Julie Lowndes, Luis López, Michele Thornton, and the NASA Openscapes Mentors. [ESIP Winter Meeting](https://2023januaryesipmeeting.sched.com/event/1EwXE/enabling-open-science-with-nasas-earthdata-in-the-cloud) “Enabling Open Science with NASA’s Earthdata in the Cloud” Session\nNASA Openscapes: Movement building with the Flywheel - March 31, 2023, Erin Robinson & Julie Lowndes. NASA Open Source Science Working Group\nShifting institutional culture to develop climate solutions with Open Science Lowndes et al 2023 (preprint, EarthArXiv)" + "objectID": "contributing/troubleshooting.html#git-update-revert-etc", + "href": "contributing/troubleshooting.html#git-update-revert-etc", + "title": "Troubleshooting", + "section": "Git: update, revert, etc", + "text": "Git: update, revert, etc\nThese are some useful commands to revert/delete your local changes and update your fork with the most recent information from the main branch.\n\nDelete your local changes\nThere are several ways to delete your local changes if you were playing around and want to reset. Here are a few:\n\nUndo changes you’ve maybe saved or committed, but not pushed\nThis is less time and internet intensive (no new clone/download).\nIf you’ve got changes saved, but not yet staged, committed, or pushed, you’ll delete unstaged changes in the working directory with clean:\nYou’ll need to make sure you’re in the github repository (use pwd to check your present working directory and cd to change directory)\ngit clean -df\ngit checkout -- .\n\n\nBurn it all down\nYou’ll delete the whole repo that you have locally, and then reclone.\nYou’ll need to make sure you’re in the github repository (use pwd to check your present working directory and cd to change directory)\nrm -rf YOUR-REPO\nHere is a whole blog on how to go back in time (walk back changes), with conceptual diagrams, command line code, and screenshots from RStudio. https://ohi-science.org/news/github-going-back-in-time\n\n\n\nUpdate local branch with remote main branch\nIf while you’re working you would like to update your local your-branch with the most recent updates on the main branch on GitHub.com, there are several ways to do this. Here’s one.\ngit checkout your-branch\ngit fetch\ngit merge origin/main" }, { - "objectID": "contributing/index.html", - "href": "contributing/index.html", - "title": "Contributing", + "objectID": "our-cookbook.html", + "href": "our-cookbook.html", + "title": "Our Cookbook", "section": "", - "text": "The NASA Openscapes Mentor Community is collaborating to create this Cookbook, with an eye towards how others could collaborate with us in the future.\nThe Openscapes Approach Guide describes how we work, including creating space and place, having a growth mindset, psychological safety, and continual learning, practice, and iteration. It also has our code of conduct and an overview of some of the workflows and tooling we use.\nWe believe open practices can accelerate data-driven solutions and increase diversity, equity, inclusion, and belonging in science. These are critical to uncovering enduring science-based solutions faster, as well as for the well-being and resilience of research communities. We work towards kinder science.\nOur style of working is greatly influenced by:\n\nThe Turing Way Community Handbook\nThe Carpentries Curriculum Development Handbook\nThe Documentation System - by Daniele Procido/Divio; see image below!\nRStudio Cheatsheets\neScience Hackweeks\n\nThe Earthdata Cloud Cookbook is made with Quarto with our 2i2c JupyterHub. Learn more in the Setup and Workflow sections.\n\n\n\nFigure from documentation.divio.com: There is a secret that needs to be understood in order to write good software documentation: there isn't one thing called documentation, there are four. They are: tutorials, how-to guides, technical reference and explanation. They represent four different purposes or functions, and require four different approaches to their creation." + "text": "Our Cookbook is a place to learn, share, and experiment with NASA Earthdata on the Cloud. We know this has a lot of moving parts, and we are iterating as we go, and welcome feedback and contributions.\nThe Cookbook has How To building blocks and Tutorials that connect these building blocks through an example research question and specific data. How To guides and Tutorials are stable approaches we’ve developed and used to teach; they have been iterated and improved through feedback from researchers during events we’ve led. We also share work In Development: primarily lessons and other works-in-progress that we’re developing.\nWorking with NASA Earthdata in the Cloud means using a combination of software, tools, many of which require coding and are unfamiliar when we first get started. This is true for us all; we’re all coming with different skills and backgrounds and you’re not alone as we all learn these new technologies and workflows. We have found it helpful to have a growth mindset - these approaches are new and we don’t know how to do them yet. Please, don’t struggle alone - know that we’re all in this together as part of the open source community learning and co-creating together as we migrate our research to the Cloud.\nYou’re able to reuse any code in this book, adapting it for your own purposes. All chapters in the book are a combination of narrative, links, code, and outputs — and underlying each chapter is a file type that you can access on NASA-Openscapes GitHub (linked on the left navbar and right margin of each page): Markdown (.md), Jupyter (.ipynb), RMarkdown (.rmd) or Quarto (.qmd) files.\nWe will also be adding citations throughout so that you can easily credit the NASA Openscapes Mentors who created them." }, { - "objectID": "contributing/index.html#how-we-work", - "href": "contributing/index.html#how-we-work", - "title": "Contributing", + "objectID": "our-cookbook.html#how-to-use-this-earthdata-cloud-cookbook", + "href": "our-cookbook.html#how-to-use-this-earthdata-cloud-cookbook", + "title": "Our Cookbook", "section": "", - "text": "The NASA Openscapes Mentor Community is collaborating to create this Cookbook, with an eye towards how others could collaborate with us in the future.\nThe Openscapes Approach Guide describes how we work, including creating space and place, having a growth mindset, psychological safety, and continual learning, practice, and iteration. It also has our code of conduct and an overview of some of the workflows and tooling we use.\nWe believe open practices can accelerate data-driven solutions and increase diversity, equity, inclusion, and belonging in science. These are critical to uncovering enduring science-based solutions faster, as well as for the well-being and resilience of research communities. We work towards kinder science.\nOur style of working is greatly influenced by:\n\nThe Turing Way Community Handbook\nThe Carpentries Curriculum Development Handbook\nThe Documentation System - by Daniele Procido/Divio; see image below!\nRStudio Cheatsheets\neScience Hackweeks\n\nThe Earthdata Cloud Cookbook is made with Quarto with our 2i2c JupyterHub. Learn more in the Setup and Workflow sections.\n\n\n\nFigure from documentation.divio.com: There is a secret that needs to be understood in order to write good software documentation: there isn't one thing called documentation, there are four. They are: tutorials, how-to guides, technical reference and explanation. They represent four different purposes or functions, and require four different approaches to their creation." + "text": "Our Cookbook is a place to learn, share, and experiment with NASA Earthdata on the Cloud. We know this has a lot of moving parts, and we are iterating as we go, and welcome feedback and contributions.\nThe Cookbook has How To building blocks and Tutorials that connect these building blocks through an example research question and specific data. How To guides and Tutorials are stable approaches we’ve developed and used to teach; they have been iterated and improved through feedback from researchers during events we’ve led. We also share work In Development: primarily lessons and other works-in-progress that we’re developing.\nWorking with NASA Earthdata in the Cloud means using a combination of software, tools, many of which require coding and are unfamiliar when we first get started. This is true for us all; we’re all coming with different skills and backgrounds and you’re not alone as we all learn these new technologies and workflows. We have found it helpful to have a growth mindset - these approaches are new and we don’t know how to do them yet. Please, don’t struggle alone - know that we’re all in this together as part of the open source community learning and co-creating together as we migrate our research to the Cloud.\nYou’re able to reuse any code in this book, adapting it for your own purposes. All chapters in the book are a combination of narrative, links, code, and outputs — and underlying each chapter is a file type that you can access on NASA-Openscapes GitHub (linked on the left navbar and right margin of each page): Markdown (.md), Jupyter (.ipynb), RMarkdown (.rmd) or Quarto (.qmd) files.\nWe will also be adding citations throughout so that you can easily credit the NASA Openscapes Mentors who created them." }, { - "objectID": "contributing/community.html", - "href": "contributing/community.html", - "title": "Community Resources", - "section": "", - "text": "Running the NASA Cloud Workshop notebooks with mybinder.org - by Eli Holmes, 2021 Cloud Hackathon Participant who then set up working in Binder" + "objectID": "our-cookbook.html#get-started", + "href": "our-cookbook.html#get-started", + "title": "Our Cookbook", + "section": "Get Started", + "text": "Get Started\nSo, you want to get started working with NASA Earthdata in the cloud? You’ve come to the right place. Here you’ll find resources that can be considered precursors to the how to’s, tutorials, and other guidance you will find across our Cookbook.\n\nEarthdata Login\nTo access NASA Earthdata, whether through your web browser or the cloud, you must first register for an Earthdata Login (EDL) user profile. Once registered, you can use your login credentials to get data through multiple access points. Read about EDL and get registered by following the directions on the Welcome to Earthdata Login page.\n\n\nCoding Essentials\nTo access the cloud programmatically we must have a basic understanding of how to code using common, cloud-relevant languages. Most scientists who work with Earth data use either Python or R already, so we focus on those languages. Python and R are both open-source programming languages widely used in web applications, software development, data science, and machine learning. They are popular because they are free, efficient, have online resources to learn, and can run on most platforms. If you are new to coding, we recommend you participate in a Carpentries Workshop or use open-source resources to teach yourself.\n\n\nBash & Git\nCloud services often are connected to and operated through Bash, a command-line interface and language you’ll see called the terminal, the command line, or the shell. Git is a commonly used version control system that is accessible through Bash. Version control is important for data collaboration because it allows changes by multiple people to be tracked and merged into one source.\n\nThe Unix Shell These lessons will introduce you the shell, a fundamental tool for performing a wide range of computing tasks.\nVersion Control with Git This tutorial will introduce you to Git, a popular open source distributed version control system.\n\n\n\nPython\n\nThe Python Tutorial This tutorial introduces the reader informally to the basic concepts and features of the Python language and system.\nPythia Foundations This book is intended to educate the reader on the essentials for using the Scientific Python Ecosystem (SPE): a collection of open source Python packages that support analysis, manipulation, and visualization of scientific data.\n\n\n\nR\n\nR for Excel Users This course is a friendly intro to becoming a modern R user, full of tidyverse, RMarkdown, GitHub, collaboration, and reproducibility.\nR for Data Science This book will teach you how to do data science with R, including how to get your data into R, get it into the most useful structure, transform it, and visualize.\n\n\n\nCloud-Native Geospatial Formats Guide\nIf you are wondering “why cloud?” and / or wish to learn more about cloud-native geospatial formats, please visit https://guide.cloudnativegeo.org/.\nFor advanced coding guidance, see our How To’s, Tutorials, and Appendix." }, { - "objectID": "in-development/matlab.html", - "href": "in-development/matlab.html", - "title": "MATLAB Access Single NASA EarthData L2 NetCDF", + "objectID": "when-to-cloud.html", + "href": "when-to-cloud.html", + "title": "When To ‘Cloud’", "section": "", - "text": "This tutorial was written by Erin Robinson, Luis Lopez (NASA Openscapes) and Cedric Wannaz, Lisa Kempler (Mathworks)." - }, - { - "objectID": "in-development/matlab.html#launch-matlab-in-2i2c-jupyterhub", - "href": "in-development/matlab.html#launch-matlab-in-2i2c-jupyterhub", - "title": "MATLAB Access Single NASA EarthData L2 NetCDF", - "section": "Launch MATLAB in 2i2c JupyterHub", - "text": "Launch MATLAB in 2i2c JupyterHub\nYou’re able to use MATLAB in the 2i2c JupyterHub; you are able to bring your own license. If you’d like to access MATLAB from the Amazon Web Services Console, see details at the bottom of this page.\n\n\n\nSelect MATLAB from the Server Options and press Start\n\n\n\n\n\nLaunch MATLAB from the JupyterHub interface by double-clicking\n\n\n\n\n\nEnter your MATLAB login information" - }, - { - "objectID": "in-development/matlab.html#credentials", - "href": "in-development/matlab.html#credentials", - "title": "MATLAB Access Single NASA EarthData L2 NetCDF", - "section": "Credentials", - "text": "Credentials\nDirect S3 access is achieved by passing NASA supplied temporary credentials to AWS so we can interact with S3 objects from applicable EarthData Cloud buckets. For now, each NASA DAAC has different AWS credentials endpoints. Below are some of the credential endpoints to various DAACs:\n\nPODAAC: https://archive.podaac.earthdata.nasa.gov/s3credentials\nGESDISC: https://data.gesdisc.earthdata.nasa.gov/s3credentials\nLPDAAC: https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials\nORNLDAAC: https://data.ornldaac.earthdata.nasa.gov/s3credentials\nGHRCDAAC: https://data.ghrc.earthdata.nasa.gov/s3credentials\nNSIDC: https://data.nsidc.earthdatacloud.nasa.gov/s3credentials\n\nYou will need your EarthData login to access these links. Create an account here: https://urs.earthdata.nasa.gov/users/new." - }, - { - "objectID": "in-development/matlab.html#prerequisites", - "href": "in-development/matlab.html#prerequisites", - "title": "MATLAB Access Single NASA EarthData L2 NetCDF", - "section": "Prerequisites", - "text": "Prerequisites\n\nSave the following function in a text file called loadAWSCredentials.txt to your MATLAB working folder or to any folder that is on MATLAB search path.\n\nfunction s3Credentials = loadAWSCredentials(daacCredentialsEndpoint, login, password)\n if nargin < 2 || isempty(login)\n login = getenv('EARTHDATA_LOGIN') ;\n end\n if nargin < 3 || isempty(password)\n password = getenv('EARTHDATA_PASSWORD') ;\n end\n\n % Get S3 credentials from EarthData\n opts = weboptions('ContentType', 'json', 'HeaderFields', ...\n {'Authorization', ['Basic ',matlab.net.base64encode([login,':',password])]});\n s3Credentials = webread(daacCredentialsEndpoint, opts) ;\n\n % Set relevant environment variables with AWS credentials/region\n setenv('AWS_ACCESS_KEY_ID', s3Credentials.accessKeyId) ;\n setenv('AWS_SECRET_ACCESS_KEY', s3Credentials.secretAccessKey) ;\n setenv('AWS_SESSION_TOKEN', s3Credentials.sessionToken) ;\n setenv('AWS_DEFAULT_REGION', 'us-west-2') ;\nend\n\nDefine environment variables that specify your EarthData credentials by executing the following (updated with your credentials):\n\nsetenv(\"EARTHDATA_LOGIN\",\"your_earthdata_login\");\nsetenv(\"EARTHDATA_PASSWORD\",\"your_earthdata_password\");\nThis can be done manually in the command window, in your MATLAB startup script, or in any MATLAB function/script that is executed before calling loadAWSCredentials. We recommend not to do it from your project main script, so you can share it without divulging your credentials.\n\nSearch NASA EarthData and find the S3 link you want to access." - }, - { - "objectID": "in-development/matlab.html#accessing-data", - "href": "in-development/matlab.html#accessing-data", - "title": "MATLAB Access Single NASA EarthData L2 NetCDF", - "section": "Accessing data", - "text": "Accessing data\nMATLAB HDF5 interfaces can access netCDF4 files stored in S3/Azure, while the netCDF interfaces cannot. The example below shows how to access a netCDF file using the HDF5 interfaces. SEe also this video: MATLAB and HDF5: Compression, Cloud, and Community – Ellen Johnson, MathWorks.\nMATLAB provides two interfaces for accessing HDF5 datasets:\n\nHigh-level access functions make it easy to read and view data in an HDF5 file or write a variable from the MATLAB® workspace into an HDF5 file\nLow-level functions in the HDF5 library packages provide direct access to the more than 300 functions in the HDF5 C library from the HDF Group.\n\nLow-level functions allow you to access advanced HDF5 functionality and bypasses I/O overhead in the high-level interface, but has a substantially higher learning curve unless already familiar with the HDF5 C API.\n\nExample\n\nFind the relevant DAAC/endpoint for the dataset that you want to access. For the following dataset:\n\ns3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20100619062008-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\nThe relevant DAAC is PODAAC and the relevant endpoint is:\nhttps://archive.podaac.earthdata.nasa.gov/s3credentials\n\nSet AWS credentials using the loadAWSCredentials function, passing the relevant endpoint:\n\ndaacCredentialsEndpoint = \"https://archive.podaac.earthdata.nasa.gov/s3credentials\";\nloadAWSCredentials(daacCredentialsEndpoint);\nThese credentials are temporary; they expire every 30 minutes or so and must be refreshed periodically.\n\nAccess data Define path of relevant file and dataset:\n\nFILE_NAME = \"s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20100619062008-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\";\nDATASET_NAME = \"sea_surface_temperature_4um\";\nDATASET_PATH = \"/\" + DATASET_NAME ;\nFirst, check that you can access the dataset:\nh5disp(FILE_PATH);\nIf credentials and paths are correct, h5disp will display the structure of the data file.\n\n\nUsing MATLAB high-level HDF5 interface\nunits = h5readatt(FILE_NAME,DATASET_PATH,\"units\");\nfill_value = h5readatt(FILE_NAME,DATASET_PATH,\"_FillValue\");\nname = h5readatt(FILE_NAME,DATASET_PATH,\"long_name\");\ndata = h5read(FILE_NAME,DATASET_PATH);\ndata(data == fill_value) = NaN;\nlat = h5read(FILE_NAME,\"/lat\");\nlon = h5read(FILE_NAME,\"/lon\");\nBuild a map using contour:\ncontour(lon,lat,data);\ntitle(sprintf(\"%s [%s]\",name,units));\n\n\n\nMATLAB example map\n\n\n\n\nUsing MATLAB low-level HDF5 interface\n% Open file and root group\nfile_id = H5F.open(FILE_NAME,\"H5F_ACC_RDONLY\",\"H5P_DEFAULT\");\nroot_id = H5G.open(file_id,\"/\") ;\n\n% Open/read sea surface temperature (SST) dataset\ndata_id = H5D.open(file_id,DATASET_PATH);\ndata=H5D.read(data_id,\"H5T_NATIVE_DOUBLE\",\"H5S_ALL\",\"H5S_ALL\",\"H5P_DEFAULT\");\n\n% Open/read/close SST units attribute\nattr_id = H5A.open_by_name(root_id,DATASET_NAME,\"units\");\nunits = H5A.read(attr_id,\"H5ML_DEFAULT\");\nH5A.close(attr_id);\n\n% Open/read/close SST long_name attribute\nattr_id = H5A.open_by_name(root_id,DATASET_NAME,\"long_name\");\nlong_name = H5A.read(attr_id,\"H5ML_DEFAULT\");\nH5A.close(attr_id);\n\n% Open/read/close SST _FillValue attribute\nattr_id = H5A.open_by_name(root_id,DATASET_NAME,\"_FillValue\");\nfillvalue = H5A.read(attr_id,\"H5T_NATIVE_DOUBLE\");\nH5A.close(attr_id);\n\n% Close SST dataset and root group\nH5D.close(data_id);\nH5G.close(root_id);\n\n% Open/read/close latitude dataset\nlat_id = H5D.open(file_id,\"lat\");\nlat = H5D.read(lat_id,\"H5T_NATIVE_DOUBLE\",\"H5S_ALL\",\"H5S_ALL\",\"H5P_DEFAULT\");\nH5D.close(lat_id);\n\n% Open/read/close longitude dataset\nlon_id = H5D.open(file_id,\"lon\");\nlon = H5D.read(lon_id,\"H5T_NATIVE_DOUBLE\",\"H5S_ALL\",\"H5S_ALL\",\"H5P_DEFAULT\");\nH5D.close(lon_id);\n\n% Close file\nH5F.close(file_id);\n\n% Replace fill value with NaN\ndata(data==fillvalue) = NaN;\n\n% Plot contour map\ncontour(lon,lat,data);\ntitle(sprintf(\"%s [%s]\",long_name{1},units{1}));" - }, - { - "objectID": "in-development/matlab.html#launch-with-aws-console", - "href": "in-development/matlab.html#launch-with-aws-console", - "title": "MATLAB Access Single NASA EarthData L2 NetCDF", - "section": "Launch with AWS Console", - "text": "Launch with AWS Console\nAn alternative is to launch with the AWS Console. This chapter is for research teams currently working in MATLAB with NASA EarthData and wanting to take advantage of doing analysis in the Cloud. Our initial focus is on Amazon Web Services (AWS). For general, background on the Cloud and NASA Earthdata’s migration to the Cloud, checkout earlier chapters of the cookbook [TO DO: Add a link and specific reference].\n\nPrerequisites\n\nMATLAB License - You must have access to a MATLAB license that allows for access to the cloud. To check your license, in the MATLAB command line enter:\nver -support\nAWS Account - For the current experiments, we will access the AWS console, so you need an AWS account. It is free to set up, but will require a credit card linked to the account. https://aws.amazon.com/\nAWS Region: We are working in AWS US-West-2 because NASA Earthdata is hosted in US-West-2. If there is an option to pick an AWS region, pick US-West-2.\nGitHub Account - If you want to use GitHub as part of your workflow and don’t already have an account, create a GitHub account: https://github.com/\nNASA Earthdata login - Create an account here: https://urs.earthdata.nasa.gov/users/new\nMicrosoft Remote Desktop - When we spin up the AWS virtual machine with MATLAB, we will need to have a remote desktop option. These how-to’s were done on a mac and we tested the free Microsoft Remote Desktop. Download for macs\n\n\n\nPrerequisite: AWS EC2 Key Pair\nThis isn’t included in the numbered prereq list above because it is a bit more involved than creating an account or knowing what region we work in. You need to create an SSH Key Pair in the region you want to work in. This is a one-time step that you won’t need to do each time you launch the stack.\n\nLog in to AWS\nIn the search box enter, ‘Create AWS Key Pair.’ Under features choose ‘Key Pairs’ (AWS help)\n\nCheck that the top bar says, ‘Oregon’ (AKA - US-West-2) and if not, click that down arrow to choose US-West-2. Then click the orange ‘Create key pair’ button.\n\nTroubleshooting tip: If you create the key outside of the region you want to work in, it will not show up when you launch the stack below.\nOn the form - give your key a name, choose the RSA and PEM options (these should be the defaults.)\n\nOk - now we are ready to start!\n\n\nCreating the AWS Stack with MATLAB\nNote: The first time I launched the AWS Stack it seemed to take for-ever. I thought “how could I possibly need to do this every time I want to use MATLAB in the cloud?” It does speed up, eventually it get’s a bit faster. I also have learned to plan a bit better - if I know I want to do some work, I get the launched, have coffee, and when I come back it’s ready to roll.\n\nFrom this MATLAB GitHub page click the release for 2022a under deployment steps.\nThis brings up a list of MATLAB on Amazon Web Services (Linux VM). Choose & click the ‘launch stack’ link for US-West-2.\n\nThis opens the ‘Quick create stack’ form based on the MATLAB template. That means that when you launch this stack it will come with MATLAB on the desktop. Fill out the form to create the AWS stack:\n\nGive the stack a name like ‘matlab-test’\nKeep the pre-filled options the same for now.\nRemote Access:\n\n“Allow Connections From:” You will need to know your IP address. You can google, “what’s my IP address?”\nEnter your IP address followed by a /32 like this -> [my.IP.address/32]\nIn the SSH Key Pair - the key pair you created above should show up in the drop down. If it doesn’t show up, see the troubleshooting tip.\nPick a remote password. This is not your AWS password or your GitHub password, this is the password that you will use to login with the microsoft remote desktop (username: ubuntu)\n\nNetwork configuration\n\nThere is one VPC option - choose that\nFor subnet - I pick the first one and it works. So pick the first option.\n\nAutoshutdown hasn’t worked for me so far, so for now I leave this set as never and delete the stack when I am finished.\nCheck the box that “I acknowledge that AWS CloudFormation might create IAM resources.”\nClick ‘Create stack’\nWait…. [~ 10 minutes]\n\nYou can check the status by clicking the refresh button on the right corner\n\n\n\n\nLaunch the AWS Stack with Microsoft Remote Desktop\n\nOnce the stack is created it will say ‘Create_complete’ on the left side.\n\nClick the outputs tab and copy the value text. It will start with ‘ec2-…’\nOpen Microsoft Remote Desktop\n\nClick the + to add a PC\nPaste the value text as the PC Name\nClick on the grey box of your new computer in the remote desktop window\nA login will pop up\n\n\nUsername is ubuntu\nPassword is the password you set in 3.3.3 above in the section on ‘Creating AWS Stack’\n\nA certificate message will pop up - say ok\nThe desktop will launch\n\nWait … [~2 mins]\n\n\n\nOpen MATLAB on Remote Desktop\n\nClick the MATLAB icon on the remote desktop\n\nWait … [~4 mins]\nLogin with your MATLAB credentials\nYou are in!\n\n\n\nShutdown your AWS Stack\nAfter each session you need to turn off the AWS Stack. If you forget this step and leave it running it is like keeping a computer on for the month. For the large instance it costs $0.5/day so it’s a few dollars a month.\n\nGo back to AWS\nSearch for stack\nClick on the name of your stack\nClick ‘Delete’\n\nConfirm the delete" + "text": "Cloud adoption often has a steep learning curve and can feel overwhelming. There are times when using the cloud is effective and times when the download model is more appropriate. Here we aim to help you decide what’s best for your use case.\n\nWhat does it mean to be in The Cloud?\n\n\n\nImage by Alexis Hunzinger, GES DISC\n\n\nAt a basic level, “The Cloud” is somewhere that isn’t your computer. We all interact with data and services and that live in “The Cloud” in our daily lives. When we store photos in iCloud or Google accounts instead of on our cell phones, we are using cloud storage. When we watch movies and tv shows on streaming services like Netflix or Hulu, we are using the cloud. In these cases, we are interacting with “the cloud” without knowing it, though we, the user, are not in “the cloud”.\n\n\n\nImage by Alexis Hunzinger, GES DISC\n\n\nIf you use services like computing and storage, provided by a cloud service provider (Amazon Web Services, Microsoft Azure, Google Cloud Platform, etc.), then you are in “the cloud”. Remember, “the cloud” is somewhere that isn’t your computer. The storage location is elsewhere and the machines and processing is elsewhere.\n\n\n\nImage by Alexis Hunzinger, GES DISC\n\n\nThe following guidance is specific to NASA Earthdata and the particular cloud it is stored in, Amazon Web Services (AWS). While some services provided by NASA data archives are cloud-based and you interact with, the guidance below refers to using compute and storage services provided by AWS that allow a user to work closely, or in the same cloud, as the data.\n\n\n\nImage by Alexis Hunzinger, GES DISC\n\n\n\n\nQuestions to ask yourself\nUse the following questions to guide your decisionmaking process about using cloud access and computing:\n\nWhat is the data volume?\nHow long will it take to download?\nCan you store all that data (cost and space)?\nDo you have the computing power for processing?\nDoes your team need a common computing environment?\nDo you need to share data at each step or just an end product?\nIs the action I want to take an existing service? (i.e. subsetting a dataset)\n\nSource: Data strategies for Future Us by Andy Barrett\n\n\nTo Cloud…\nTODO: list example cases here: identify and describe what makes cloud usage more benficial for the team/workflow\nFind more discussion on cloud-based solutions for geoscience in this Medium article by Ryan Abernathy and Joe Hamman.\n\n\nNot To Cloud…\nTODO: list examples cases here: identify and describe what makes the non-option more beneficial for the team/workflow\n\n\nChallenges\nIt is important to be aware of the drawbacks and challenges associated with working in the cloud. Feedback from early cloud adopters are summarized here:\n\n“I don’t have the time or energy to learn so many new concepts.”\n“Major challenge: budgeting for cloud in a proposal.”\n“My workflow isn’t parallelizable, the file format isn’t cloud-optimized, I’d rather download everything I need and perform analysis locally.”\n“[On the preference to perform analysis locally] …this is fine if data wasn’t getting bigger!”\n“[The cloud] is faster, but only by a factor of 2. Our setup and data granules aren’t structured to take advantage of a faster cloud.”\n“Worried about doing things ‘on my own’, outside of a cloud-hosted JupyterHub.”\n“How does using data from AWS work with GEE?”\n\nSource: The Cloud: Obstacles and Barriers Encountered by Users (AGU 2023 - Alexis Hunzinger, Christopher Battisto, Allison Alcott, Binita KC)\n\n\nConsiderations\nWe are now accustomed to living in a highly digital world, separated from physical reminders of the technology we use. No longer do we access documents from a row of filing cabinets, we now store them in cloud-based archives (e.g. Google Docs). We run code on high-performance computing clusters, removed from the whirring and warmth generated by servers that live away from our desks. The following are considerations for using powerful cloud-based resources:\n\nRemoved from physical signs of energy usage (heat, noise) of supercomputers/servers\nPhysical location of “the cloud”, whose land is it on? What resources is it using?\nEnvironmental impacts of large server farms\nConsider testing locally before migrating a workflow to the cloud\n\nDoes cloud storage and computing increase equity?\n\nYes! Cloud is a solution for equity, as it provides under-resourced organizations (e.g. small universities, international groups) access to equipment they don’t own or maintain.\nNo! Larger organizations, who already own and maintain equipment for storage and computing, often have the budget to also use cloud storage and computing. This increases their pool of resources, giving them a larger advantage." }, { "objectID": "in-development/index.html", @@ -2457,725 +2303,697 @@ "text": "These lessons are in development; as they are developed further and tested they will move into other parts of the Cookbook!" }, { - "objectID": "tutorials/fair-workflow-geoweaver-demo.html", - "href": "tutorials/fair-workflow-geoweaver-demo.html", - "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", + "objectID": "in-development/CMR-STAC-Search.html", + "href": "in-development/CMR-STAC-Search.html", + "title": "Getting Started with NASA’s CMR-STAC API in R", "section": "", - "text": "PyGeoweaver package is a Python wrapper of the GeoWeaver. This package is designed for Jupyter users to be able to directly use Geoweaver in Jupyter notebook or JupyterLab (JupyterHub). ## 1. Installing PyGeoweaver You can either install pygeoweaver using pip or build from source\n!pip install pygeoweaver\n\nRequirement already satisfied: pygeoweaver in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (0.7.4)\nRequirement already satisfied: requests==2.28.2 in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (from pygeoweaver) (2.28.2)\nRequirement already satisfied: setuptools>=61.0 in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (from pygeoweaver) (68.0.0)\nRequirement already satisfied: pydantic==1.10.9 in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (from pygeoweaver) (1.10.9)\nRequirement already satisfied: typing-extensions>=4.2.0 in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (from pydantic==1.10.9->pygeoweaver) (4.6.3)\nRequirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (from requests==2.28.2->pygeoweaver) (1.26.16)\nRequirement already satisfied: certifi>=2017.4.17 in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (from requests==2.28.2->pygeoweaver) (2023.5.7)\nRequirement already satisfied: idna<4,>=2.5 in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (from requests==2.28.2->pygeoweaver) (3.4)\nRequirement already satisfied: charset-normalizer<4,>=2 in /Users/l21-n02609-comm/opt/anaconda3/lib/python3.9/site-packages (from requests==2.28.2->pygeoweaver) (3.1.0)\nThe gw.start() function starts the Geoweaver Application on background to serve requests\nimport pygeoweaver as gw\ngw.start()\n\nStop running Geoweaver if any..\nCheck Java..\nStart Geoweaver..\nSuccess: Geoweaver is up" - }, - { - "objectID": "tutorials/fair-workflow-geoweaver-demo.html#reset-passwords-optional-if-you-already-did", - "href": "tutorials/fair-workflow-geoweaver-demo.html#reset-passwords-optional-if-you-already-did", - "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", - "section": "2. Reset Passwords (optional if you already did)", - "text": "2. Reset Passwords (optional if you already did)\nThis password will be used to authenticate the process runs on localhost (a.k.a the default host with id 100001).\n\ngw.reset_password()\n\nEnter password: ········\nRe-enter password: ········\n\n\nNOTE: Password updated." - }, - { - "objectID": "tutorials/fair-workflow-geoweaver-demo.html#import-workflow-earthdata-cloud-clinic", - "href": "tutorials/fair-workflow-geoweaver-demo.html#import-workflow-earthdata-cloud-clinic", - "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", - "section": "3. Import Workflow : Earthdata Cloud Clinic", - "text": "3. Import Workflow : Earthdata Cloud Clinic\nThis tutorial from Cloud cookbook uses two different direct cloud access & subsetting options available in the Earthdata Cloud. * The earthaccess python library for data search and direct cloud access, followed by xarray subsetting * The Harmony-py python library for direct cloud access & data subsetting\nThe workflow for earthdata cloud clinic for geoweaver is hosted at https://github.com/earth-artificial-intelligence/earth_data_cloud_clinic_geoweaver\n\nimport os\nimport urllib.request\nfrom IPython.display import Image\n\nhome_dir = os.path.expanduser('~')\nfile_url = \"https://github.com/earth-artificial-intelligence/earth_data_cloud_clinic_geoweaver/archive/refs/tags/v0.0.6.zip\"\nfile_path = os.path.join(home_dir, \"kr2novmm4i9i27z5adyj.zip\")\n\nurllib.request.urlretrieve(file_url, file_path)\n\ngw.import_workflow(file_path)\n\nUnzipping /Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj.zip to /Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj/\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/.DS_Store\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/README.md\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/earth_access_authenticate.py\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/earth_data_utils.py\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/install_deps.sh\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/install_packages.sh\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/process.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/retrieve_ssh.py\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/retrieve_sst.py\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/code/s3_file_storage.py\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/earthdata.ipynb\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/geoweaver_screenshot.png\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/gulf.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/195yXqgSfvPIInJ2gI.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/197a10n0xbrc4rqwve.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/1bZvPF5Z1tytBGDqph.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/39SB7eVc2494ECqik2.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/3KpWI4XE8npoDNFXTQ.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/3ZlO4QZpqjhGyKmyXO.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/43kckrxzh5QbESss0R.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/54jcaiqoso3iree437.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/5dixki8g2scz0sjj56.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/5ou0GVSM1NQnlv7FOB.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/5ucpoaa3e0bbnwyu1n.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/5uzqmkhig32ub23cax.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/6LDOLdZroopQ1UvaWF.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/6hxs1emptzyrmsi0f3.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/8CmlmhGPYckTE5VQnh.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/8nqne2wcyke9t1hy5p.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/Ay4VYNLteCA4TvHHvt.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/B2kMBsZQW3Adxgi1uH.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/BacPAagztwKFogMSWa.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/IbB3mgAjkrED9qC8zw.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/JUM7mZ2Clo86FSJQpS.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/KeJgbxTIefDdZTHTLf.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/KmoUyTblH4DeCQYfWN.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/PNFURuqn5MBZk110Fh.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/QBaTZsjXUzZrYyaSGY.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/TX1TxAhpPjO6dvySl7.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/UbPSHj43zEmCk2k5CR.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/UfT7rXlmUDYmfoP804.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/VeCiz4t1MvIQX4i2yN.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/YBoSD2A5E8pi15Utoi.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/YZGTgiuACdQqM2CObt.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/Z9cOHlCTrfGR4beaLN.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/ZYvkLM99T33qWDzU5l.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/a4t3g4kcmx2j5iggfu.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/b70aawjt8z4isn1dwy.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/b8oEcCvvIU6KaKiYoT.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/bv0wvvajfbrscyj6pm.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/c3lYCsxly0BDlEg6da.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/cf2KWtlaFRUAoEIhAS.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/cn80gdwbo9cmtqfmt4.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/csltVkAvLqfza0jV0i.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/drP4XgYr92dISzae7n.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/e02dq17h7tng220fo1.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/eiXb07m8PuodHs3y9W.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/exp8ncff5wdbitkm8o.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/fbSMCmb9pAMK1bogT0.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/gh2rmy5cq1bc003ak4.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/h26w0736xpdftw6zw1.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/hCz5UKbmmsYoiPeelF.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/iMtwJlVtkG2jGPEJCm.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/jid4vegdc9y0pz1oco.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/ju768g303se38lkt74.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/kX9THeiXUlECn7qnLK.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/kialj0m6av03b04sv3.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/kr2novmm4i9i27z5adyj.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/kr6AXjoMUHFXn74s6B.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/l6yVTXJZi87L8UPfDB.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/l873z9b33henwzrydx.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/lXakuW2z080MDGe8i1.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/lbTt2RL9gtcLVP6xKj.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/lwpewb4927s4e4m49r.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/lxABhFBxbAvzawPiKs.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/mczeu09JHbwVkuAAeC.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/mr9YcFRUnuNyThJb4Q.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/mx1w8k46bha4zi0q8r.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/n4tc1hvnq5eamswpkj.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/nfuxH8eVz896NHRm1m.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/nkx236238gagh5wycu.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/nnyv3p5uhplqv43ml3.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/o0enoj5tgqss95igeb.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/oU0aQLscHakCzO9BOi.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/or6xD9foI8celpyyrB.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/otrr4bjr5vvwob8kzf.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/p5qpQWqza1Fwv0GbsF.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/pYWDe5CUpDNQ9g08Uw.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_1b4ko0.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_2a8mok.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_cgvhiy.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_e57twj.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_nlsntt.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_qmc7xi.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_ru79i2.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_v2ow93.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_wttkvt.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/process_yvo1ds.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/rRrz0ESWwjNUfd8J4l.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/sRPxDerwGuaBIYhhAO.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/tFbeUeqkws6xyNbueE.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/tXEbbqVKQhjVLLM2f3.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/u6EibxNEZg5Or4d4gZ.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/unlenty6p8qrl03z42.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/v97kvdt3cfum1172y2.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/vC36VuDH7D6OOMF68w.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/vx911STRa1BDoIfOd7.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/wDqF0CmSx2MfoE037z.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/woFrPp1c1EdP5vQQMr.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/wq5bex7m1w1t185yyw.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/wxJMPLZFgS9gqClzF6.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/xU3NHiBe4cXiTolHSR.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/xdc6fso7wkybrk71o7.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/xsUiKqCumWr86iaEFQ.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/xvrhhmdem4lsrax6io.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/y2mnxktf5j68wxy68b.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/yf0JCKUqg9sfzi0Tlq.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/ywrvvwo20iif52rga6.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/zkammrkv9md6gesze8.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/history/zuwTMUYPw7YXrBjWxj.json\n/Users/joe/gw-workspace/temp/kr2novmm4i9i27z5adyj - earth_data_cloud_clinic_geoweaver-0.0.6/workflow.json\nPrecheck: Done\nWorkflow ID: kr2novmm4i9i27z5adyj\nSuccessfully saved to database. Complete." - }, - { - "objectID": "tutorials/fair-workflow-geoweaver-demo.html#check-workflow-details", - "href": "tutorials/fair-workflow-geoweaver-demo.html#check-workflow-details", - "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", - "section": "4. Check Workflow Details", - "text": "4. Check Workflow Details\nThe workflow has four processes:\n\ninstall_packages - install required packages\nearth_data_utils - define all the utility functions like auth_earthaccess, get_dataset by xarray, etc\nretrieve_ssh - using earthaccess to query EarthDataCloud for SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205\nretrieve_sst - using earthaccess to query EarthDataCloud for MUR-JPL-L4-GLOB-v4.1\n\n\n\n\nScreen Shot 2023-06-21 at 12.41.43 AM.png\n\n\nCheck workflow id and other metadata using detail_workflow.\n\ngw.detail_workflow(\"kr2novmm4i9i27z5adyj\")\n\nWorkflowId: kr2novmm4i9i27z5adyj\n+----------------------+-------------------------+-------------+--------+---------------+\n| Workflow Id | Name | Description | Owner | Confidential? |\n+----------------------+-------------------------+-------------+--------+---------------+\n| kr2novmm4i9i27z5adyj | earth_data_cloud_clinic | | 111111 | FALSE |\n+----------------------+-------------------------+-------------+--------+---------------+" - }, - { - "objectID": "tutorials/fair-workflow-geoweaver-demo.html#run-workflow", - "href": "tutorials/fair-workflow-geoweaver-demo.html#run-workflow", - "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", - "section": "5. Run workflow", - "text": "5. Run workflow\nThe follow line will execute the workflow on the provided host. 100001 means localhost. You can use gw.detail_host(\"100001\") to check the details.\nIt will prompt a password input field to ask for password of the target host. If you don’t know your default password of localhost, please run gw.reset_password().\n\ngw.run_workflow(workflow_id='kr2novmm4i9i27z5adyj', host_list='100001')\n\nEnter password for host - 100001: ········\n\n\nRunning workflow kr2novmm4i9i27z5adyj\nThe workflow has been kicked off.\nHistory Id: svx32gpq6x3xmqkorm\nWaiting for it to finish\n> Start to run process: ru79i2\n> Fnished process: ru79i2 - history: zg6v1hy042k - Status: Done\n> Start to run process: e57twj\n> Fnished process: e57twj - history: o2qf88jyh2f - Status: Done\n> Start to run process: nlsntt\n> Fnished process: nlsntt - history: fk604fwceda - Status: Done\nTotal time cost: 104 seconds\nExecution is over. Final status: Done." - }, - { - "objectID": "tutorials/fair-workflow-geoweaver-demo.html#check-workflow-history", - "href": "tutorials/fair-workflow-geoweaver-demo.html#check-workflow-history", - "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", - "section": "6. Check workflow history", - "text": "6. Check workflow history\nFrom the log, we can see the workflow completed successfully after running all the four member processes. The final process run’s history id is ohvdmvpfy1m. We can use show_history to check.\n\n## Get Results by Checking History\ngw.show_history(\"fk604fwceda\")\n\n+-------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n| History Id | Status | Begin Time | End Time | Input | Output | Notes |\n+-------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n| fk604fwceda | Done | 06/21/2023 03:30:26 | 06/21/2023 03:31:00 | from earth_data_utils import *\n\ndef search_and_get_sst():\n auth_earthaccess()\n results = earthaccess.search_data(\n short_name=sst_short_name,\n cloud_hosted=True,\n temporal=(\"2021-07-01\", \"2021-07-02\"),\n )\n print(\"found results: \", results)\n return results\n\ndef plot_subset(ds_subset):\n ds_subset.plot(figsize=(12,6), x='lon', y='lat')\n home_dir = os.path.expanduser('~')\n \n file_name = f\"geoweaver_plot_sst.png\"\n save_file_path = os.path.join(home_dir, file_name)\n plt.savefig(save_file_path)\n print(f\"new figure is saved to {save_file_path}\")\n\nwith get_dataset(search_and_get_sst()) as ds:\n subset = ds[\"analysed_sst\"].sel(lat=slice(27, 37), \n lon=slice(-70,-50)).std('time')\n plot_subset(subset)\n\n | You're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 08/19/2023\nUsing .netrc file for EDL\nGranules found: 2\nfound results: [Collection: {'Version': '4.1', 'ShortName': 'MUR-JPL-L4-GLOB-v4.1'}\nSpatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'BoundingRectangles': [{'WestBoundingCoordinate': -180, 'SouthBoundingCoordinate': -90, 'EastBoundingCoordinate': 180, 'NorthBoundingCoordinate': 90}]}}}\nTemporal coverage: {'RangeDateTime': {'EndingDateTime': '2021-07-01T21:00:00.000Z', 'BeginningDateTime': '2021-06-30T21:00:00.000Z'}}\nSize(MB): 0\nData: ['https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20210701090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc'], Collection: {'Version': '4.1', 'ShortName': 'MUR-JPL-L4-GLOB-v4.1'}\nSpatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'BoundingRectangles': [{'WestBoundingCoordinate': -180, 'SouthBoundingCoordinate': -90, 'EastBoundingCoordinate': 180, 'NorthBoundingCoordinate': 90}]}}}\nTemporal coverage: {'RangeDateTime': {'EndingDateTime': '2021-07-02T21:00:00.000Z', 'BeginningDateTime': '2021-07-01T21:00:00.000Z'}}\nSize(MB): 0\nData: ['https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20210702090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc']]\n Opening 2 granules, approx size: 0.0 GB\nQUEUEING TASKS | : 0%| | 0/2 [00:00<?, ?it/s]\nQUEUEING TASKS | : 100%|██████████| 2/2 [00:00<00:00, 1421.56it/s]\nPROCESSING TASKS | : 0%| | 0/2 [00:00<?, ?it/s]\nPROCESSING TASKS | : 50%|█████ | 1/2 [00:00<00:00, 1.51it/s]\nPROCESSING TASKS | : 100%|██████████| 2/2 [00:00<00:00, 2.97it/s]\nPROCESSING TASKS | : 100%|██████████| 2/2 [00:00<00:00, 2.59it/s]\nCOLLECTING RESULTS | : 0%| | 0/2 [00:00<?, ?it/s]\nCOLLECTING RESULTS | : 100%|██████████| 2/2 [00:00<00:00, 22192.08it/s]\nnew figure is saved to /Users/joe/geoweaver_plot_sst.png\n | - |\n+-------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n\n\nFrom the history, we can see the result image location is at /Users/joe/geoweaver_plot_sst.png. Let’s display it here." - }, - { - "objectID": "tutorials/fair-workflow-geoweaver-demo.html#display-sea-surface-temperature-data", - "href": "tutorials/fair-workflow-geoweaver-demo.html#display-sea-surface-temperature-data", - "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", - "section": "7. Display sea surface temperature data", - "text": "7. Display sea surface temperature data\n\ndisplay(Image(filename=os.path.join(home_dir, 'geoweaver_plot_sst.png')))\n\n\n\n\nBingo! The image is successfully retrieved by the workflow and the data in xarray is rendered and saved to a local png file. The workflow runs with no errors and the replication succeeded." + "text": "This tutorial demonstrates how to interact with CMR-STAC in R.\nThis tutorial will teach you how to navigate and explore NASA’s Common Metadata Repository (CMR) SpatioTemporal Asset Catalog (STAC) to learn about the datasets available through LP DAAC Cumulus cloud archive." }, { - "objectID": "tutorials/fair-workflow-geoweaver-demo.html#edit-the-process-to-change-date-range", - "href": "tutorials/fair-workflow-geoweaver-demo.html#edit-the-process-to-change-date-range", - "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", - "section": "8. Edit the process to change date range", - "text": "8. Edit the process to change date range\nReplicating is only the start. We always want to change the code and make it work for our purposes. Here we will demonstrate how to change the code in the workflow.\nFirst, let’s list all the processes in the workflow:\n\ngw.list_processes_in_workflow(workflow_id='kr2novmm4i9i27z5adyj')\n\n\n\n\n\n\n\n\ntitle\nid\n\n\n\n\n0\nearth_data_utils\nwttkvt-GJjq8\n\n\n1\ninstall_packages\nru79i2-VCNCv\n\n\n2\nretrieve_sst\nnlsntt-zMUba\n\n\n3\nretrieve_ssh\ne57twj-ECmak\n\n\n\n\n\n\n\nYou can see the ids are a little different with dashes in the middle of them. That is because we need to distinguish the processes in the workflow as there might be same processes called multiple times in one workflow.\nThe original process id is just the first part. For example, the process id of retrieve_sst is nlsntt.\nCopy that id and use get_code_for_process function to get current code.\n\ngw.get_process_code('nlsntt')\n\n'from earth_data_utils import *\\n\\ndef search_and_get_sst():\\n auth_earthaccess()\\n results = earthaccess.search_data(\\n short_name=sst_short_name,\\n cloud_hosted=True,\\n temporal=(\"2021-07-01\", \"2021-07-02\"),\\n )\\n print(\"found results: \", results)\\n return results\\n\\ndef plot_subset(ds_subset):\\n ds_subset.plot(figsize=(12,6), x=\\'lon\\', y=\\'lat\\')\\n home_dir = os.path.expanduser(\\'~\\')\\n \\n file_name = f\"geoweaver_plot_sst.png\"\\n save_file_path = os.path.join(home_dir, file_name)\\n plt.savefig(save_file_path)\\n print(f\"new figure is saved to {save_file_path}\")\\n\\nwith get_dataset(search_and_get_sst()) as ds:\\n subset = ds[\"analysed_sst\"].sel(lat=slice(27, 37), \\n lon=slice(-70,-50)).std(\\'time\\')\\n plot_subset(subset)\\n\\n'\n\n\nThe code is not formatted correctly. So many \\n are confusing.\nAn easy way is to directly save the code into a local file using function sync.\n\n# write the file to disk to make changes\ngw.sync(process_id='nlsntt', local_path=\".\", direction=\"download\")\n\nProceeding with download\n\nWrote file retrieve_sst.py to .\n\n\nThe step above should create a new python file named retrieve_sst.py in the same folder as this notebook.\nOpen it using any editor and you should see the code as follows:\nfrom earth_data_utils import *\n\ndef search_and_get_sst():\n auth_earthaccess()\n results = earthaccess.search_data(\n short_name=sst_short_name,\n cloud_hosted=True,\n temporal=(\"2021-07-01\", \"2021-07-02\"),\n )\n print(\"found results: \", results)\n return results\n\ndef plot_subset(ds_subset):\n ds_subset.plot(figsize=(12,6), x='lon', y='lat')\n home_dir = os.path.expanduser('~')\n \n file_name = f\"geoweaver_plot_{str(uuid.uuid4())}.png\"\n save_file_path = os.path.join(home_dir, file_name)\n plt.savefig(save_file_path)\n print(f\"new figure is saved to {save_file_path}\")\n\nwith get_dataset(search_and_get_sst()) as ds:\n subset = ds[\"analysed_sst\"].sel(lat=slice(27, 37), \n lon=slice(-70,-50)).std('time')\n plot_subset(subset)\nSuppose we want the SST data of another region, for example, change the lat=slice(27, 37) to lat=slice(37, 47). Make the change in retrieve_sst.py, save and close.\nNext step is to write the new code back into Geoweaver process using the same sync function. This time, the option direction should be upload meaning we are syncing changes from local file to Geoweaver database.\n\n# sync changes back to the workflow after editing the file\ngw.sync(process_id='nlsntt', local_path='retrieve_sst.py', direction=\"upload\")\n\nProceeding with upload\n\nProcess update was successful\n\n\nAfter the step finishes, verify the code change using get_process_code again.\n\ngw.get_process_code('nlsntt')\n\n'from earth_data_utils import *\\n\\ndef search_and_get_sst():\\n auth_earthaccess()\\n results = earthaccess.search_data(\\n short_name=sst_short_name,\\n cloud_hosted=True,\\n temporal=(\"2021-07-01\", \"2021-07-02\"),\\n )\\n print(\"found results: \", results)\\n return results\\n\\ndef plot_subset(ds_subset):\\n ds_subset.plot(figsize=(12,6), x=\\'lon\\', y=\\'lat\\')\\n home_dir = os.path.expanduser(\\'~\\')\\n \\n file_name = f\"geoweaver_plot_sst.png\"\\n save_file_path = os.path.join(home_dir, file_name)\\n plt.savefig(save_file_path)\\n print(f\"new figure is saved to {save_file_path}\")\\n\\nwith get_dataset(search_and_get_sst()) as ds:\\n subset = ds[\"analysed_sst\"].sel(lat=slice(37, 47), \\n lon=slice(-70,-50)).std(\\'time\\')\\n plot_subset(subset)\\n\\n'\n\n\nOk, great! You can see the changes are made as the lat= has become 37, 47 now. The editing process is complete.\nNow, we just need to rerun the workflow again with the new code.\n\ngw.run_workflow(workflow_id='kr2novmm4i9i27z5adyj', host_list='100001')\n\nEnter password for host - 100001: ········\n\n\nRunning workflow kr2novmm4i9i27z5adyj\nThe workflow has been kicked off.\nHistory Id: cf5xnsjounspuhlk9z\nWaiting for it to finish\n> Start to run process: ru79i2\n> Fnished process: ru79i2 - history: arr51wopzbw - Status: Done\n> Start to run process: e57twj\n> Fnished process: e57twj - history: ppyfc2he31g - Status: Done\n> Start to run process: nlsntt\n> Fnished process: nlsntt - history: e1ce5zjdm3y - Status: Done\nTotal time cost: 105 seconds\nExecution is over. Final status: Done.\n\n\nYou can see there is a new history id for process nlsntt: m3t1sdcsl3k. Check its content to find the result png file location:\n\ngw.show_history(\"e1ce5zjdm3y\")\n\n+-------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n| History Id | Status | Begin Time | End Time | Input | Output | Notes |\n+-------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n| e1ce5zjdm3y | Done | 06/21/2023 03:37:55 | 06/21/2023 03:38:33 | from earth_data_utils import *\n\ndef search_and_get_sst():\n auth_earthaccess()\n results = earthaccess.search_data(\n short_name=sst_short_name,\n cloud_hosted=True,\n temporal=(\"2021-07-01\", \"2021-07-02\"),\n )\n print(\"found results: \", results)\n return results\n\ndef plot_subset(ds_subset):\n ds_subset.plot(figsize=(12,6), x='lon', y='lat')\n home_dir = os.path.expanduser('~')\n \n file_name = f\"geoweaver_plot_sst.png\"\n save_file_path = os.path.join(home_dir, file_name)\n plt.savefig(save_file_path)\n print(f\"new figure is saved to {save_file_path}\")\n\nwith get_dataset(search_and_get_sst()) as ds:\n subset = ds[\"analysed_sst\"].sel(lat=slice(37, 47), \n lon=slice(-70,-50)).std('time')\n plot_subset(subset)\n\n | You're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 08/19/2023\nUsing .netrc file for EDL\nGranules found: 2\nfound results: [Collection: {'Version': '4.1', 'ShortName': 'MUR-JPL-L4-GLOB-v4.1'}\nSpatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'BoundingRectangles': [{'WestBoundingCoordinate': -180, 'SouthBoundingCoordinate': -90, 'EastBoundingCoordinate': 180, 'NorthBoundingCoordinate': 90}]}}}\nTemporal coverage: {'RangeDateTime': {'EndingDateTime': '2021-07-01T21:00:00.000Z', 'BeginningDateTime': '2021-06-30T21:00:00.000Z'}}\nSize(MB): 0\nData: ['https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20210701090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc'], Collection: {'Version': '4.1', 'ShortName': 'MUR-JPL-L4-GLOB-v4.1'}\nSpatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'BoundingRectangles': [{'WestBoundingCoordinate': -180, 'SouthBoundingCoordinate': -90, 'EastBoundingCoordinate': 180, 'NorthBoundingCoordinate': 90}]}}}\nTemporal coverage: {'RangeDateTime': {'EndingDateTime': '2021-07-02T21:00:00.000Z', 'BeginningDateTime': '2021-07-01T21:00:00.000Z'}}\nSize(MB): 0\nData: ['https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20210702090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc']]\n Opening 2 granules, approx size: 0.0 GB\nQUEUEING TASKS | : 0%| | 0/2 [00:00<?, ?it/s]\nQUEUEING TASKS | : 100%|██████████| 2/2 [00:00<00:00, 1568.26it/s]\nPROCESSING TASKS | : 0%| | 0/2 [00:00<?, ?it/s]\nPROCESSING TASKS | : 50%|█████ | 1/2 [00:00<00:00, 1.21it/s]\nPROCESSING TASKS | : 100%|██████████| 2/2 [00:00<00:00, 2.42it/s]\nCOLLECTING RESULTS | : 0%| | 0/2 [00:00<?, ?it/s]\nCOLLECTING RESULTS | : 100%|██████████| 2/2 [00:00<00:00, 21454.24it/s]\nnew figure is saved to /Users/joe/geoweaver_plot_sst.png\n | - |\n+-------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n\n\nThe new png is generated. Visualize it in jupyter.\n\ndisplay(Image(filename=os.path.join(home_dir, 'geoweaver_plot_sst.png')))\n\n\n\n\nNice! The figure looks great with SST texture different from the one displayed above. That means the change of the spatial extent in the process worked great!\nThe reproducing experiment is done." + "objectID": "in-development/CMR-STAC-Search.html#a.-what-is-stac", + "href": "in-development/CMR-STAC-Search.html#a.-what-is-stac", + "title": "Getting Started with NASA’s CMR-STAC API in R", + "section": "1a. What is STAC?", + "text": "1a. What is STAC?\nSTAC is short for Spatiotemporal Asset Catalog, a series of specifications that provide a common language for interpreting geospatial information in order to standardize indexing and discovery of spatiotemporal assets (files containing information about the Earth across space and time).\nThere are four specifications that work both independently and together:\n\nSTAC Catalog\nSTAC Collection\nSTAC Item\n\nSTAC API specification builds on top of the three core specifications mentioned above. All these specifications are intended to be used together, yet are designed in a way that each piece is small, self-contained, and reusable in other contexts." }, { - "objectID": "tutorials/fair-workflow-geoweaver-demo.html#check-history-of-specific-processes-or-workflows", - "href": "tutorials/fair-workflow-geoweaver-demo.html#check-history-of-specific-processes-or-workflows", - "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", - "section": "9. Check history of specific processes or workflows", - "text": "9. Check history of specific processes or workflows\n\n# get run history of the process\ngw.get_process_history('nlsntt')\n\n\n\n\n\n\n\n\nhistory_id\nhistory_input\nhistory_output\nhistory_begin_time\nhistory_end_time\nhistory_notes\nhistory_process\nhost_id\nindicator\n\n\n\n\n0\ne1ce5zjdm3y\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 07:37:55.840\n2023-06-21 07:38:33.137\nNone\nnlsntt\n100001\nDone\n\n\n1\nafz7j3dzavr\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 07:33:45.739\n2023-06-21 07:34:17.617\nNone\nnlsntt\n100001\nDone\n\n\n2\nfk604fwceda\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 07:30:26.691\n2023-06-21 07:31:00.588\nNone\nnlsntt\n100001\nDone\n\n\n3\nuf7ngwa1cnl\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 07:27:23.479\n2023-06-21 07:27:56.093\nNone\nnlsntt\n100001\nDone\n\n\n4\njxyo4sjnw7i4dsvt30\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 07:07:44.443\n2023-06-21 07:08:22.248\nNone\nnlsntt\nNone\nDone\n\n\n5\nm3t1sdcsl3k\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 07:02:58.529\n2023-06-21 07:03:31.363\nNone\nnlsntt\n100001\nDone\n\n\n6\nohvdmvpfy1m\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 04:35:07.128\n2023-06-21 04:35:41.058\nNone\nnlsntt\n100001\nDone\n\n\n7\n55ierovrm5x\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 04:21:51.164\n2023-06-21 04:22:26.277\nNone\nnlsntt\n100001\nDone\n\n\n8\nEcOKe1e6zRlz\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 04:19:36.446\n2023-06-21 04:20:10.986\nNone\nnlsntt\nNone\nDone\n\n\n9\nClldd9m98PAt\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 04:15:33.330\n2023-06-21 04:16:09.377\nNone\nnlsntt\nNone\nDone\n\n\n10\nsAZF2bohiT4g\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 04:10:58.009\n2023-06-21 04:11:29.683\nNone\nnlsntt\nNone\nDone\n\n\n11\nmuhcxxqhzna\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 04:08:44.419\n2023-06-21 04:09:36.507\nNone\nnlsntt\n100001\nDone\n\n\n12\n5c4g1wdrzme\nfrom earth_data_utils import *\\n\\ndef search_a...\nRunning\n2023-06-21 04:07:23.881\n2023-06-21 04:07:35.184\nNone\nnlsntt\n100001\nDone\n\n\n13\nV4V7YE0Q20Qd\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 04:05:26.906\n2023-06-21 04:06:20.076\nNone\nnlsntt\nNone\nDone\n\n\n14\ngH0Y1ntFvvJQ\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 04:01:01.589\n2023-06-21 04:01:57.329\nNone\nnlsntt\nNone\nDone\n\n\n15\nbc8wLVHggQC2\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 03:55:43.750\n2023-06-21 03:56:34.311\nNone\nnlsntt\nNone\nFailed\n\n\n16\nfFuSBuGyhfUh\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 03:44:26.103\n2023-06-21 03:55:39.992\nNone\nnlsntt\nNone\nDone\n\n\n17\nMt9qs1sz3ypu\nfrom earth_data_utils import *\\n\\ndef search_a...\nFile \"/Users/joe/gw-workspace/Mt9qs1sz3ypu/r...\n2023-06-21 03:44:13.505\n2023-06-21 03:44:13.553\nNone\nnlsntt\nNone\nFailed\n\n\n18\ng5wgNPIMKvt6\nfrom earth_data_utils import *\\n\\ndef search_a...\nFile \"/Users/joe/gw-workspace/g5wgNPIMKvt6/r...\n2023-06-21 03:43:54.668\n2023-06-21 03:43:54.724\nNone\nnlsntt\nNone\nFailed\n\n\n19\nR1pmcdWqLp6V\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 03:40:48.248\n2023-06-21 03:41:17.846\nNone\nnlsntt\nNone\nFailed\n\n\n20\nD5poheZLB3st\nfrom earth_data_utils import *\\n\\ndef search_a...\nTraceback (most recent call last):\\n File \"/U...\n2023-06-21 03:40:35.390\n2023-06-21 03:40:41.692\nNone\nnlsntt\nNone\nFailed\n\n\n21\n5lqwvos3wb6\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 03:33:59.859\n2023-06-21 03:34:27.842\nNone\nnlsntt\n100001\nFailed\n\n\n22\nXYWhBOUtZcPG\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 03:25:02.150\n2023-06-21 03:25:36.884\nNone\nnlsntt\nNone\nFailed\n\n\n23\n4NoU0dDwnwmb\nfrom earth_data_utils import *\\n\\ndef search_a...\nTraceback (most recent call last):\\n File \"/U...\n2023-06-21 03:24:10.798\n2023-06-21 03:24:17.863\nNone\nnlsntt\nNone\nFailed\n\n\n24\n4AheIx9gmzVw\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 03:15:01.301\n2023-06-21 03:15:33.319\nNone\nnlsntt\nNone\nFailed\n\n\n25\nGNDa18SBhJTV\nfrom earth_data_utils import *\\n\\ndef search_a...\nYou're now authenticated with NASA Earthdata L...\n2023-06-21 03:11:51.951\n2023-06-21 03:12:27.523\nNone\nnlsntt\nNone\nFailed\n\n\n\n\n\n\n\n\ngw.get_workflow_history('kr2novmm4i9i27z5adyj')\n\n\n\n\n\n\n\n\nhistory_id\nhistory_input\nhistory_output\nhistory_begin_time\nhistory_end_time\nhistory_notes\nhistory_process\nhost_id\nindicator\n\n\n\n\n0\ncf5xnsjounspuhlk9z\nwttkvt-GJjq8;ru79i2-VCNCv;nlsntt-zMUba;e57twj-...\nfwu7qad3kq6;arr51wopzbw;e1ce5zjdm3y;ppyfc2he31g;\n2023-06-21 07:36:46.153\n2023-06-21 07:37:55.840000000\nNone\nkr2novmm4i9i27z5adyj\n100001;\nDone\n\n\n1\ndad8g3gkb76stetwix\nwttkvt-GJjq8;ru79i2-VCNCv;nlsntt-zMUba;e57twj-...\n1c5sjmcrgnj;fx19g9buif9;afz7j3dzavr;jlmxs448gc7;\n2023-06-21 07:32:27.062\n2023-06-21 07:33:45.739000064\nNone\nkr2novmm4i9i27z5adyj\n100001;\nDone\n\n\n2\nsvx32gpq6x3xmqkorm\nwttkvt-GJjq8;ru79i2-VCNCv;nlsntt-zMUba;e57twj-...\nzle2vcmqtkh;zg6v1hy042k;fk604fwceda;o2qf88jyh2f;\n2023-06-21 07:29:18.002\n2023-06-21 07:30:26.691000064\nNone\nkr2novmm4i9i27z5adyj\n100001;\nDone\n\n\n3\nb70aawjt8z4isn1dwy\nwttkvt-GJjq8;ru79i2-VCNCv;nlsntt-zMUba;e57twj-...\n617pj1d8nis;zzcxtngjbdu;uf7ngwa1cnl;agb7rrec49u;\n2023-06-21 07:26:09.740\n2023-06-21 07:26:14.760999936\nNone\nkr2novmm4i9i27z5adyj\n100001;\nRunning\n\n\n4\nbv0wvvajfbrscyj6pm\nwttkvt-GJjq8;ru79i2-VCNCv;nlsntt-zMUba;e57twj-...\n1h6mwhjgffr;ced80h6vr3r;m3t1sdcsl3k;ed1lczm7cgh;\n2023-06-21 07:01:45.800\n2023-06-21 07:02:58.528999936\nNone\nkr2novmm4i9i27z5adyj\n100001;\nDone\n\n\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n\n\n94\nnfuxH8eVz896NHRm1m\nwttkvt-GJjq8;yvo1ds-96vns;cgvhiy-BgnXM;v2ow93-...\nrm0v06cmvuc;8nq18ogdwdi;jhzwkhohmz7;6o2p0699ri...\n2023-06-08 16:08:37.776\n2023-06-08 16:08:43.665999872\nNone\nkr2novmm4i9i27z5adyj\njf7wuu;\nFailed\n\n\n95\nVeCiz4t1MvIQX4i2yN\nwttkvt-GJjq8;yvo1ds-96vns;cgvhiy-BgnXM;v2ow93-...\n15jtnmmliz8;sj2llldbybq;alm4h4h5irw;fs47znp6yg...\n2023-06-08 16:08:06.441\n2023-06-08 16:08:14.723000064\nNone\nkr2novmm4i9i27z5adyj\njf7wuu;\nFailed\n\n\n96\ntFbeUeqkws6xyNbueE\nwttkvt-GJjq8;yvo1ds-96vns;cgvhiy-BgnXM;v2ow93-...\n5gi96g5psgp;7ikdfl3nkyj;zftnvonktj6;aipjqmvzj5...\n2023-06-08 16:06:07.658\nNaT\nNone\nkr2novmm4i9i27z5adyj\njf7wuu;\nStopped\n\n\n97\nPNFURuqn5MBZk110Fh\nwttkvt-GJjq8;yvo1ds-96vns;cgvhiy-BgnXM;v2ow93-...\nro72gckii7a;dnqed48moze;xb7v9iap72l;8heewks755...\n2023-06-08 16:05:50.800\nNaT\nNone\nkr2novmm4i9i27z5adyj\njf7wuu;\nStopped\n\n\n98\nwxJMPLZFgS9gqClzF6\nwttkvt-GJjq8;yvo1ds-96vns;cgvhiy-BgnXM;v2ow93-...\n9a06gtkzws7;4z8l2w2pq5n;2z9k699i4su;w8f5ftv7f9...\n2023-06-08 16:05:15.441\nNaT\nNone\nkr2novmm4i9i27z5adyj\njf7wuu;\nStopped\n\n\n\n\n99 rows × 9 columns" + "objectID": "in-development/CMR-STAC-Search.html#b.-what-is-the-cmr-stac-api", + "href": "in-development/CMR-STAC-Search.html#b.-what-is-the-cmr-stac-api", + "title": "Getting Started with NASA’s CMR-STAC API in R", + "section": "1b. What is the CMR-STAC API?", + "text": "1b. What is the CMR-STAC API?\nThe Common Metadata Repository (CMR) is a metadata system that catalogs Earth Science data and associated metadata records. NASA’s CMR-STAC Application Programming Interface (API) is a translation API for STAC users who want to access and search through CMR’s vast metadata holdings using STAC keywords." }, { - "objectID": "tutorials/fair-workflow-geoweaver-demo.html#directly-run-local-python-process-using-pygeoweaver", - "href": "tutorials/fair-workflow-geoweaver-demo.html#directly-run-local-python-process-using-pygeoweaver", - "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", - "section": "10. Directly run local python process using PyGeoweaver", - "text": "10. Directly run local python process using PyGeoweaver\n\ngw.run_process(host_id='100001', process_id='nlsntt', sync_path='retrieve_sst.py')\n\nEnter password for host - 100001: ········\n\n\nUpdating code on workflow with the given file path.\n\nStaring process nlsntt\nExecution finished\nTotal time cost: 41 seconds\n+--------------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n| History Id | Status | Begin Time | End Time | Input | Output | Notes |\n+--------------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n| orznud2v2ugsyqfjlj | Done | 06/21/2023 03:40:26 | 06/21/2023 03:41:00 | from earth_data_utils import *\n\ndef search_and_get_sst():\n auth_earthaccess()\n results = earthaccess.search_data(\n short_name=sst_short_name,\n cloud_hosted=True,\n temporal=(\"2021-07-01\", \"2021-07-02\"),\n )\n print(\"found results: \", results)\n return results\n\ndef plot_subset(ds_subset):\n ds_subset.plot(figsize=(12,6), x='lon', y='lat')\n home_dir = os.path.expanduser('~')\n \n file_name = f\"geoweaver_plot_sst.png\"\n save_file_path = os.path.join(home_dir, file_name)\n plt.savefig(save_file_path)\n print(f\"new figure is saved to {save_file_path}\")\n\nwith get_dataset(search_and_get_sst()) as ds:\n subset = ds[\"analysed_sst\"].sel(lat=slice(37, 47), \n lon=slice(-70,-50)).std('time')\n plot_subset(subset)\n\n | You're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 08/19/2023\nUsing .netrc file for EDL\nGranules found: 2\nfound results: [Collection: {'Version': '4.1', 'ShortName': 'MUR-JPL-L4-GLOB-v4.1'}\nSpatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'BoundingRectangles': [{'WestBoundingCoordinate': -180, 'SouthBoundingCoordinate': -90, 'EastBoundingCoordinate': 180, 'NorthBoundingCoordinate': 90}]}}}\nTemporal coverage: {'RangeDateTime': {'EndingDateTime': '2021-07-01T21:00:00.000Z', 'BeginningDateTime': '2021-06-30T21:00:00.000Z'}}\nSize(MB): 0\nData: ['https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20210701090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc'], Collection: {'Version': '4.1', 'ShortName': 'MUR-JPL-L4-GLOB-v4.1'}\nSpatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'BoundingRectangles': [{'WestBoundingCoordinate': -180, 'SouthBoundingCoordinate': -90, 'EastBoundingCoordinate': 180, 'NorthBoundingCoordinate': 90}]}}}\nTemporal coverage: {'RangeDateTime': {'EndingDateTime': '2021-07-02T21:00:00.000Z', 'BeginningDateTime': '2021-07-01T21:00:00.000Z'}}\nSize(MB): 0\nData: ['https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20210702090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc']]\n Opening 2 granules, approx size: 0.0 GB\nQUEUEING TASKS | : 0%| | 0/2 [00:00<?, ?it/s]\nQUEUEING TASKS | : 100%|██████████| 2/2 [00:00<00:00, 2267.80it/s]\nPROCESSING TASKS | : 0%| | 0/2 [00:00<?, ?it/s]\nPROCESSING TASKS | : 50%|█████ | 1/2 [00:00<00:00, 1.53it/s]\nPROCESSING TASKS | : 100%|██████████| 2/2 [00:00<00:00, 2.71it/s]\nCOLLECTING RESULTS | : 0%| | 0/2 [00:00<?, ?it/s]\nCOLLECTING RESULTS | : 100%|██████████| 2/2 [00:00<00:00, 16131.94it/s]\nnew figure is saved to /Users/joe/geoweaver_plot_sst.png\n | - |\n+--------------------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+\n\n\nUse the same code to visualize the result in a plot.\n\ndisplay(Image(filename=os.path.join(home_dir, 'geoweaver_plot_sst.png')))\n\n\n\n\nThis image has the same texture as we got in the previous workflow run, and it means running local python directly in pygeoweaver will have the same outputs and won’t impact the consistency.\nPyGeoWeaver offers significant benefits over manual management of geospatial processes. It ensures that result images remain consistent regardless of file location changes, enables seamless integration with your local Python file, and provides a repeatable and automated process. By utilizing PyGeoWeaver, you can achieve consistent, reproducible, and efficient geospatial analysis, enhancing your productivity and reducing the burden of manual workflow management." + "objectID": "in-development/CMR-STAC-Search.html#a.-cmr-stac-api", + "href": "in-development/CMR-STAC-Search.html#a.-cmr-stac-api", + "title": "Getting Started with NASA’s CMR-STAC API in R", + "section": "2a. CMR-STAC API", + "text": "2a. CMR-STAC API\nThe CMR-STAC API contains endpoints that enable the querying of STAC items.\nAssign the CMR-STAC URL to a static variable.\n\nCMR_STAC_URL <- 'https://cmr.earthdata.nasa.gov/stac/'\n\nConnect to the CMR-STAC landing page which contains all the available data providers and their STAC endpoint. In this tutorial, the httr package is used to navigate CMR-STAC API.\n\ncmr_cat <- httr::GET(CMR_STAC_URL) %>% # Request and retrieve the info from CMR-STAC URL\n httr::content() \ncat('You are using',cmr_cat$title,'version',cmr_cat$stac_version,\".\", cmr_cat$description,sep=\" \")\n\nYou are using NASA CMR STAC Proxy version 1.0.0 . This is the landing page for CMR-STAC. Each provider link contains a STAC endpoint.\n\n\nHere, jsonlite is used to change the format of the content returned from our request and the DT package is used to make the returned information more readable. The providers’ names and URL links are found in the title and ‘href’ fields respectively.\n\ncmr_cat_links <- cmr_cat$links %>% \n jsonlite::toJSON(auto_unbox = TRUE) %>% \n jsonlite::fromJSON() %>% \n as.data.frame()\nDT::datatable(cmr_cat_links)\n\n\n\n\n\n\nThe data frame above shows all the data providers with their associated STAC catalog endpoints. You will notice above that the CMR-STAC API contains many different endpoints–not just from NASA LP DAAC, but also contains endpoints for other NASA ESDIS DAACs. Use the title field to identify the data provider you are interested in. The data product used in this tutorial is hosted in the LP DAAC Cumulus Cloud space (LPCLOUD).\nAssign LPCLOUD to the provider variable and get this provider’s endpoint from the CMR catalog using the URL in Link field.\n\nprovider <- 'LPCLOUD'\nlpcloud_cat_link <- cmr_cat_links[which(cmr_cat_links$title == provider), 'href']\nlpcloud_cat_link\n\n[1] \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD\"" }, { - "objectID": "tutorials/fair-workflow-geoweaver-demo.html#geoweaver-gui-for-creating-and-editing-workflows", - "href": "tutorials/fair-workflow-geoweaver-demo.html#geoweaver-gui-for-creating-and-editing-workflows", - "title": "Openscapes Cohort 2023 - Geoweaver FAIRable and Tangible Workflow Demo", - "section": "11. Geoweaver GUI for creating and editing workflows", - "text": "11. Geoweaver GUI for creating and editing workflows\nThere is something not covered in this tutorial about how to create a new workflow in Geoweaver. We usually do that in the Geoweaver GUI.\nCheck if you can see Geoweaver GUI using the show method like the picture below.\n\n\n\nimage.png\n\n\nIf you’re running Jupyter on your localhost, accessing Geoweaver is like a breeze. If you’re on your localhost, simply type gw.show in a Jupyter cell block or navigate to http://localhost:8070/Geoweaver in your browser, and you’re good to go. However, if you’re running Jupyter on a remote host, a little extra setup is needed to access Geoweaver.\nTo access Geoweaver on a remote host, you’ll need to proxy the URL to a public address. For instance, if you’re using an AWS EC2 instance, replace “localhost” in the URL with the public IP address of your instance. Then enter the modified URL in your browser. Don’t forget to add port 8070 to the inbound rule in your EC2 instance’s security group settings to ensure a smooth connection.\nIf you’re eager to explore Geoweaver’s GUI and dive into detailed tutorials, you can find them at https://esipfed.github.io/Geoweaver/. However, in this notebook, our focus is on demonstrating the incredible capabilities of Geoweaver for replicating and reproducing workflows.\n\ngw.show()\n\n2023-06-21 03:17:27,586 - INFO - enter ipython block\n2023-06-21 03:17:27,587 - WARNING - This only works when the Jupyter is visited from localhost!" + "objectID": "in-development/CMR-STAC-Search.html#b.-stac-catalog", + "href": "in-development/CMR-STAC-Search.html#b.-stac-catalog", + "title": "Getting Started with NASA’s CMR-STAC API in R", + "section": "2b. STAC Catalog", + "text": "2b. STAC Catalog\nSTAC Catalog Contains a JSON file of links that organize all the available collections. Below, connect to the LPCLOUD STAC Catalog endpoint using httr package and print the information contained in the Catalog.\n\nlpcloud_cat <- httr::GET(lpcloud_cat_link) %>% \n httr::content()\n\nlpcloud_cat <- lpcloud_cat %>% \n jsonlite::toJSON(auto_unbox = TRUE) %>% \n jsonlite::fromJSON() \n\nDT::datatable(lpcloud_cat$links)\n\n\n\n\n\n\nLPCLOUD STAC catalog includes URL links to the root, collections, search, and child STAC Catalogs. The data frame above also shows the available collections in the LPCLOUD catalog." }, { - "objectID": "tutorials/Sea_Level_Rise.html", - "href": "tutorials/Sea_Level_Rise.html", - "title": "Analyzing Sea Level Rise with NASA Earthdata", - "section": "", - "text": "image.png" + "objectID": "in-development/CMR-STAC-Search.html#c.-stac-collection", + "href": "in-development/CMR-STAC-Search.html#c.-stac-collection", + "title": "Getting Started with NASA’s CMR-STAC API in R", + "section": "2c. STAC Collection", + "text": "2c. STAC Collection\nSTAC Collection is extension of STAC Catalog containing additional information that describe the STAC Items in that Collection.\nGet the URL link to the STAC Collections.\n\nlpcloud_col_link <- lpcloud_cat$links[which(lpcloud_cat$links$rel == 'collections'),'href']\nlpcloud_col_link\n\n[1] \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections\"\n\n\nNext, get the content describing the collections within LPCLOUD Catalog. Important information such as data collection ID, title, description, and links to collection endpoints are provided here.\n\nlpcloud_collection <- httr::GET(lpcloud_col_link) %>% \n httr::content() \n\nlpcloud_collection <- lpcloud_collection %>% \n jsonlite::toJSON(auto_unbox = TRUE, pretty = TRUE)\n\nPrint the collections within LPCLOUD STAC catalog.\n\nlpcloud_collection_df <- jsonlite::fromJSON(lpcloud_collection)$collections\nlpcloud_collection_df$id\n\n [1] \"ASTGTM.v003\" \"ECO_L1B_ATT.v002\" \"ECO_L2_CLOUD.v002\"\n [4] \"ECO_L1B_GEO.v002\" \"ECO_L2_LSTE.v002\" \"ECO_L1B_RAD.v002\" \n [7] \"ECO_L2T_LSTE.v002\" \"EMITL1BRAD.v001\" \"EMITL2ARFL.v001\" \n[10] \"HLSL30.v2.0\" \n\n\nIn CMR, Collection ID is used to query by a specific product, so be sure to save the ID for a collection you are interested in. For instance, the Collection ID for ASTER Global Digital Elevation Model V003 is ASTGTM.v003. Note that the “id” shortname is in the format: productshortname.vVVV (where VVV = product version).\nHere, get the URL link to the ASTGTM.v003 STAC Collection. If you are interested in querying a different LPCLOUD product, swap out the shortname to assign to the collection variable below.\n\ncollection <- 'ASTGTM.v003' # USER INPUT\ncol_links <- lpcloud_collection_df$links[which(lpcloud_collection_df$id == collection)] %>% \n as.data.frame()\n\nastgtm_URL <- col_links[which(col_links$rel == 'self'), 'href']\nastgtm_URL\n\n[1] \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/ASTGTM.v003\"\n\n\nThe STAC Collection metadata for any collection contains metadata and information that is applicable to every STAC Item and asset(s) that it contains. Get the content of the ASTGTM.v003 collection URL and print the collection description.\n\nastgtm_collection <- httr::GET(astgtm_URL) %>% \n httr::content()\n\nastgtm_collection <- astgtm_collection %>% \n jsonlite::toJSON(auto_unbox = TRUE) %>% \n jsonlite::fromJSON()\n\ncat(astgtm_collection$description)\n\nThe ASTER Global Digital Elevation Model (GDEM) Version 3 (ASTGTM) provides a global digital elevation model (DEM) of land areas on Earth at a spatial resolution of 1 arc second (approximately 30 meter horizontal posting at the equator).\n\nThe development of the ASTER GDEM data products is a collaborative effort between National Aeronautics and Space Administration (NASA) and Japan’s Ministry of Economy, Trade, and Industry (METI). The ASTER GDEM data products are created by the Sensor Information Laboratory Corporation (SILC) in Tokyo. \n\nThe ASTER GDEM Version 3 data product was created from the automated processing of the entire ASTER Level 1A (https://doi.org/10.5067/ASTER/AST_L1A.003) archive of scenes acquired between March 1, 2000, and November 30, 2013. Stereo correlation was used to produce over one million individual scene based ASTER DEMs, to which cloud masking was applied. All cloud screened DEMs and non-cloud screened DEMs were stacked. Residual bad values and outliers were removed. In areas with limited data stacking, several existing reference DEMs were used to supplement ASTER data to correct for residual anomalies. Selected data were averaged to create final pixel values before partitioning the data into 1 degree latitude by 1 degree longitude tiles with a one pixel overlap. To correct elevation values of water body surfaces, the ASTER Global Water Bodies Database (ASTWBD) (https://doi.org/10.5067/ASTER/ASTWBD.001) Version 1 data product was also generated. \n\nThe geographic coverage of the ASTER GDEM extends from 83° North to 83° South. Each tile is distributed in GeoTIFF format and projected on the 1984 World Geodetic System (WGS84)/1996 Earth Gravitational Model (EGM96) geoid. Each of the 22,912 tiles in the collection contain at least 0.01% land area. \n\nProvided in the ASTER GDEM product are layers for DEM and number of scenes (NUM). The NUM layer indicates the number of scenes that were processed for each pixel and the source of the data.\n\nWhile the ASTER GDEM Version 3 data products offer substantial improvements over Version 2, users are advised that the products still may contain anomalies and artifacts that will reduce its usability for certain applications. \n\nImprovements/Changes from Previous Versions \n• Expansion of acquisition coverage to increase the amount of cloud-free input scenes from about 1.5 million in Version 2 to about 1.88 million scenes in Version 3.\n• Separation of rivers from lakes in the water body processing. \n• Minimum water body detection size decreased from 1 km2 to 0.2 km2. \n\n\nWe can also get the spatial and temporal extent information. Below, we can see this collection has a global spatial extent. ASTER GDEM is a single, static dataset that incorporates observation from March 2000 to November 2013.\n\nastgtm_collection$extent %>% \n jsonlite::toJSON(auto_unbox = TRUE)\n\n{\"spatial\":{\"bbox\":[[-180,-83,180,82]]},\"temporal\":{\"interval\":[[\"2000-03-01T00:00:00.000Z\",\"2013-11-30T23:59:59.999Z\"]]}} \n\n\nSTAC collection also includes useful links. You can visit all the items within this collection using the Items URL.\n\nDT::datatable(astgtm_collection$links)\n\n\n\n\n\n\nGet the URL to the ASTGTM.v003 Items.\n\nitems_url <- astgtm_collection$links [which(astgtm_collection$links$rel == 'items'), 'href']\nitems_url\n\n[1] \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/ASTGTM.v003/items\"" }, { - "objectID": "tutorials/Sea_Level_Rise.html#overview", - "href": "tutorials/Sea_Level_Rise.html#overview", - "title": "Analyzing Sea Level Rise with NASA Earthdata", - "section": "Overview", - "text": "Overview\nThis tutorial analyzes global sea level rise from satellite altimetry data and compares it to a historic record. We will be reproducing the plot from this infographic: NASA-led Study Reveals the Causes of Sea Level Rise Since 1900." + "objectID": "in-development/CMR-STAC-Search.html#d.-stac-item", + "href": "in-development/CMR-STAC-Search.html#d.-stac-item", + "title": "Getting Started with NASA’s CMR-STAC API in R", + "section": "2d. STAC Item", + "text": "2d. STAC Item\nSTAC Item represents data and metadata assets that are spatiotemporally coincident. Below, query the STAC Items within the ASTGTM.v003 STAC Collection and print the first item in the collection.\n\nastgtm_items <- httr::GET(items_url) %>% \n httr::content(as = \"text\") %>% \n jsonlite::fromJSON()\n\nF1 <- astgtm_items$features[1,] %>% \n jsonlite::toJSON(auto_unbox = TRUE, pretty = TRUE)\nF1\n\n[\n {\n \"type\": \"Feature\",\n \"id\": \"ASTGTMV003_N03E008\",\n \"stac_version\": \"1.0.0\",\n \"stac_extensions\": [],\n \"collection\": \"ASTGTM.v003\",\n \"geometry\": {\n \"type\": \"Polygon\",\n \"coordinates\": [\n [\n [7.9999, 2.9999],\n [9.0001, 2.9999],\n [9.0001, 4.0001],\n [7.9999, 4.0001],\n [7.9999, 2.9999]\n ]\n ]\n },\n \"bbox\": [7.9999, 2.9999, 9.0001, 4.0001],\n \"links\": [\n {\n \"rel\": \"self\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/ASTGTM.v003/items/ASTGTMV003_N03E008\"\n },\n {\n \"rel\": \"parent\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/ASTGTM.v003\"\n },\n {\n \"rel\": \"collection\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/ASTGTM.v003\"\n },\n {\n \"rel\": \"root\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/\"\n },\n {\n \"rel\": \"provider\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD\"\n },\n {\n \"rel\": \"via\",\n \"href\": \"https://cmr.earthdata.nasa.gov/search/concepts/G1716133754-LPCLOUD.json\"\n },\n {\n \"rel\": \"via\",\n \"href\": \"https://cmr.earthdata.nasa.gov/search/concepts/G1716133754-LPCLOUD.umm_json\"\n }\n ],\n \"properties\": {\n \"datetime\": \"2000-03-01T00:00:00.000Z\",\n \"start_datetime\": \"2000-03-01T00:00:00.000Z\",\n \"end_datetime\": \"2013-11-30T23:59:59.000Z\"\n },\n \"assets\": {\n \"003/ASTGTMV003_N03E008_dem\": {\n \"href\": \"https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/ASTGTM.003/ASTGTMV003_N03E008_dem.tif\",\n \"title\": \"Download ASTGTMV003_N03E008_dem.tif\"\n },\n \"003/ASTGTMV003_N03E008_num\": {\n \"href\": \"https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/ASTGTM.003/ASTGTMV003_N03E008_num.tif\",\n \"title\": \"Download ASTGTMV003_N03E008_num.tif\"\n },\n \"browse\": {\n \"href\": \"https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/ASTGTM.003/ASTGTMV003_N03E008.1.jpg\",\n \"type\": \"image/jpeg\",\n \"title\": \"Download ASTGTMV003_N03E008.1.jpg\"\n },\n \"metadata\": {\n \"href\": \"https://cmr.earthdata.nasa.gov/search/concepts/G1716133754-LPCLOUD.xml\",\n \"type\": \"application/xml\"\n },\n \"003/ASTGTMV003_N02E022_dem\": {},\n \"003/ASTGTMV003_N02E022_num\": {},\n \"003/ASTGTMV003_N00W065_dem\": {},\n \"003/ASTGTMV003_N00W065_num\": {},\n \"003/ASTGTMV003_N01E009_dem\": {},\n \"003/ASTGTMV003_N01E009_num\": {},\n \"003/ASTGTMV003_N02E009_dem\": {},\n \"003/ASTGTMV003_N02E009_num\": {},\n \"003/ASTGTMV003_N03E021_dem\": {},\n \"003/ASTGTMV003_N03E021_num\": {},\n \"003/ASTGTMV003_N01E021_dem\": {},\n \"003/ASTGTMV003_N01E021_num\": {},\n \"003/ASTGTMV003_N01E042_dem\": {},\n \"003/ASTGTMV003_N01E042_num\": {},\n \"003/ASTGTMV003_N01W069_dem\": {},\n \"003/ASTGTMV003_N01W069_num\": {},\n \"003/ASTGTMV003_N01W080_dem\": {},\n \"003/ASTGTMV003_N01W080_num\": {}\n }\n }\n] \n\n\nNotice that the number of items matching our request is far more than what is returned.\n\ncat(astgtm_items$context$matched, 'items matched your request but', astgtm_items$context$returned, 'items are returned.')\n\n22912 items matched your request but 10 items are returned.\n\n\nThis is because the return is paginated. The STAC API, by default, returns the first 10 records. To explore more items, you can add ?page=n (in which n is the page number (i.e. ?page=2)) to the URL link and submit another request. Below, request a query to return records on the second page.\n\npage_2_url <- paste0(items_url, '?page=2')\n\nastgtm_page2_items <- httr::GET(page_2_url) %>% \n httr::content(as = \"text\") %>% \n jsonlite::fromJSON()\n\nastgtm_page2_items$features[1,] %>% \n jsonlite::toJSON(auto_unbox = TRUE, pretty = TRUE)\n\n[\n {\n \"type\": \"Feature\",\n \"id\": \"ASTGTMV003_N03E042\",\n \"stac_version\": \"1.0.0\",\n \"stac_extensions\": [],\n \"collection\": \"ASTGTM.v003\",\n \"geometry\": {\n \"type\": \"Polygon\",\n \"coordinates\": [\n [\n [41.9999, 2.9999],\n [43.0001, 2.9999],\n [43.0001, 4.0001],\n [41.9999, 4.0001],\n [41.9999, 2.9999]\n ]\n ]\n },\n \"bbox\": [41.9999, 2.9999, 43.0001, 4.0001],\n \"links\": [\n {\n \"rel\": \"self\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/ASTGTM.v003/items/ASTGTMV003_N03E042\"\n },\n {\n \"rel\": \"parent\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/ASTGTM.v003\"\n },\n {\n \"rel\": \"collection\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/ASTGTM.v003\"\n },\n {\n \"rel\": \"root\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/\"\n },\n {\n \"rel\": \"provider\",\n \"href\": \"https://cmr.earthdata.nasa.gov/stac/LPCLOUD\"\n },\n {\n \"rel\": \"via\",\n \"href\": \"https://cmr.earthdata.nasa.gov/search/concepts/G1726373735-LPCLOUD.json\"\n },\n {\n \"rel\": \"via\",\n \"href\": \"https://cmr.earthdata.nasa.gov/search/concepts/G1726373735-LPCLOUD.umm_json\"\n }\n ],\n \"properties\": {\n \"datetime\": \"2000-03-01T00:00:00.000Z\",\n \"start_datetime\": \"2000-03-01T00:00:00.000Z\",\n \"end_datetime\": \"2013-11-30T23:59:59.000Z\"\n },\n \"assets\": {\n \"003/ASTGTMV003_N03E042_dem\": {\n \"href\": \"https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/ASTGTM.003/ASTGTMV003_N03E042_dem.tif\",\n \"title\": \"Download ASTGTMV003_N03E042_dem.tif\"\n },\n \"003/ASTGTMV003_N03E042_num\": {\n \"href\": \"https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/ASTGTM.003/ASTGTMV003_N03E042_num.tif\",\n \"title\": \"Download ASTGTMV003_N03E042_num.tif\"\n },\n \"browse\": {\n \"href\": \"https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/ASTGTM.003/ASTGTMV003_N03E042.1.jpg\",\n \"type\": \"image/jpeg\",\n \"title\": \"Download ASTGTMV003_N03E042.1.jpg\"\n },\n \"metadata\": {\n \"href\": \"https://cmr.earthdata.nasa.gov/search/concepts/G1726373735-LPCLOUD.xml\",\n \"type\": \"application/xml\"\n },\n \"003/ASTGTMV003_N00W061_dem\": {},\n \"003/ASTGTMV003_N00W061_num\": {},\n \"003/ASTGTMV003_N02W066_dem\": {},\n \"003/ASTGTMV003_N02W066_num\": {},\n \"003/ASTGTMV003_N02W069_dem\": {},\n \"003/ASTGTMV003_N02W069_num\": {},\n \"003/ASTGTMV003_N01E022_dem\": {},\n \"003/ASTGTMV003_N01E022_num\": {},\n \"003/ASTGTMV003_N01E026_dem\": {},\n \"003/ASTGTMV003_N01E026_num\": {},\n \"003/ASTGTMV003_N02W064_dem\": {},\n \"003/ASTGTMV003_N02W064_num\": {},\n \"003/ASTGTMV003_N01W064_dem\": {},\n \"003/ASTGTMV003_N01W064_num\": {},\n \"003/ASTGTMV003_N01E027_dem\": {},\n \"003/ASTGTMV003_N01E027_num\": {},\n \"003/ASTGTMV003_N00E006_dem\": {},\n \"003/ASTGTMV003_N00E006_num\": {}\n }\n }\n]" }, { - "objectID": "tutorials/Sea_Level_Rise.html#learning-objectives", - "href": "tutorials/Sea_Level_Rise.html#learning-objectives", - "title": "Analyzing Sea Level Rise with NASA Earthdata", - "section": "Learning Objectives", - "text": "Learning Objectives\n\nSearch for data programmatically using keywords\nAccess data within the AWS cloud using the earthaccess python library\nVisualize sea level rise trends from altimetry datasets and compare against historic records" + "objectID": "in-development/CMR-STAC-Search.html#e.-assets", + "href": "in-development/CMR-STAC-Search.html#e.-assets", + "title": "Getting Started with NASA’s CMR-STAC API in R", + "section": "2e. Assets", + "text": "2e. Assets\nThe STAC Item ID (CMR Granule ID) is the unique identifier assigned to each granule within a data collection. Within each STAC Item are assets, which include the downloadable and streamable URL to data files along with other asset objects. Below, the first Granule ID is used to get the downloadable data file.\n\nitems_df <- jsonlite::fromJSON(F1) \nitem <- items_df$assets # Get the assets for the first Item\nassets <- purrr::map_df(items_df$assets, data.frame, .id = 'asset')\nassets\n\n asset\n1 003/ASTGTMV003_N03E008_dem\n2 003/ASTGTMV003_N03E008_num\n3 browse\n4 metadata\n5 003/ASTGTMV003_N02E022_dem\n6 003/ASTGTMV003_N02E022_num\n7 003/ASTGTMV003_N00W065_dem\n8 003/ASTGTMV003_N00W065_num\n9 003/ASTGTMV003_N01E009_dem\n10 003/ASTGTMV003_N01E009_num\n11 003/ASTGTMV003_N02E009_dem\n12 003/ASTGTMV003_N02E009_num\n13 003/ASTGTMV003_N03E021_dem\n14 003/ASTGTMV003_N03E021_num\n15 003/ASTGTMV003_N01E021_dem\n16 003/ASTGTMV003_N01E021_num\n17 003/ASTGTMV003_N01E042_dem\n18 003/ASTGTMV003_N01E042_num\n19 003/ASTGTMV003_N01W069_dem\n20 003/ASTGTMV003_N01W069_num\n21 003/ASTGTMV003_N01W080_dem\n22 003/ASTGTMV003_N01W080_num\n href\n1 https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/ASTGTM.003/ASTGTMV003_N03E008_dem.tif\n2 https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/ASTGTM.003/ASTGTMV003_N03E008_num.tif\n3 https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/ASTGTM.003/ASTGTMV003_N03E008.1.jpg\n4 https://cmr.earthdata.nasa.gov/search/concepts/G1716133754-LPCLOUD.xml\n5 <NA>\n6 <NA>\n7 <NA>\n8 <NA>\n9 <NA>\n10 <NA>\n11 <NA>\n12 <NA>\n13 <NA>\n14 <NA>\n15 <NA>\n16 <NA>\n17 <NA>\n18 <NA>\n19 <NA>\n20 <NA>\n21 <NA>\n22 <NA>\n title type\n1 Download ASTGTMV003_N03E008_dem.tif <NA>\n2 Download ASTGTMV003_N03E008_num.tif <NA>\n3 Download ASTGTMV003_N03E008.1.jpg image/jpeg\n4 <NA> application/xml\n5 <NA> <NA>\n6 <NA> <NA>\n7 <NA> <NA>\n8 <NA> <NA>\n9 <NA> <NA>\n10 <NA> <NA>\n11 <NA> <NA>\n12 <NA> <NA>\n13 <NA> <NA>\n14 <NA> <NA>\n15 <NA> <NA>\n16 <NA> <NA>\n17 <NA> <NA>\n18 <NA> <NA>\n19 <NA> <NA>\n20 <NA> <NA>\n21 <NA> <NA>\n22 <NA> <NA>\n\n\nThe links found in the href field can be used to download each specific asset." }, { - "objectID": "tutorials/Sea_Level_Rise.html#requirements", - "href": "tutorials/Sea_Level_Rise.html#requirements", - "title": "Analyzing Sea Level Rise with NASA Earthdata", - "section": "Requirements", - "text": "Requirements\n\n1. Compute environment - This notebook can only be run in an AWS Cloud instance running in the us-west-2 region.\n\n\n2. Earthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\nInstall Packages\n\nimport earthaccess\nimport xarray as xr\nfrom pyproj import Geod\nimport numpy as np\nimport hvplot.xarray\nfrom matplotlib import pyplot as plt\nfrom pprint import pprint\n\n\n\n\n\n\n\n\n\n\n\nWe recommend authenticating your Earthdata Login (EDL) information using the earthaccess python package as follows:\n\nauth = earthaccess.login(strategy=\"netrc\") # works if the EDL login already been persisted to a netrc\nif not auth.authenticated:\n # ask for EDL credentials and persist them in a .netrc file\n auth = earthaccess.login(strategy=\"interactive\", persist=True)\n\nYou're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 09/10/2023\nUsing .netrc file for EDL" + "objectID": "in-development/CMR-STAC-Search.html#a.-define-search-parameters", + "href": "in-development/CMR-STAC-Search.html#a.-define-search-parameters", + "title": "Getting Started with NASA’s CMR-STAC API in R", + "section": "3a. Define Search Parameters", + "text": "3a. Define Search Parameters\nNext, define the search parameters. - Query by collection: Collection IDs should be defined as a list. - Spatial Querying via Bounding Box: A bounding box including the coordinates of LL (lower left) and UR (upper right) respectively. - Temporal Querying: Time period of interest should be specified as YYYY-MM-DDTHH:MM:SSZ/YYYY-MM-DDTHH:MM:SSZ.\n\ncollections <- list('ASTGTM.v003')\ndatetime <- '2000-01-01T00:00:00Z/2001-01-31T23:59:59Z' #YYYY-MM-DDTHH:MM:SSZ/YYYY-MM-DDTHH:MM:SSZ\nbbox <- '-122.0622682571411,39.897234301806,-122.04918980598451,39.91309383703065' # LL and UR Coordinates\n\nCreate search body object from our search parameters.\n\nbody <- list(limit=100,\n datetime=datetime,\n bbox= bbox,\n collections= collections)\n\nNotice the limit parameter in the body object. This parameter allows us to adjust the number of records returned during a request (default = 10)." }, { - "objectID": "tutorials/Sea_Level_Rise.html#search-for-sea-surface-height-data", - "href": "tutorials/Sea_Level_Rise.html#search-for-sea-surface-height-data", - "title": "Analyzing Sea Level Rise with NASA Earthdata", - "section": "Search for Sea Surface Height Data", - "text": "Search for Sea Surface Height Data\nLet’s find the first four collections that match a keyword search for Sea Surface Height and print out the first two.\n\ncollections = earthaccess.collection_query().keyword(\"SEA SURFACE HEIGHT\").cloud_hosted(True).get(4)\n\n# Let's print 2 collections\nfor collection in collections[0:2]:\n # pprint(collection.summary())\n print(pprint(collection.summary()), collection.abstract(), \"\\n\", collection[\"umm\"][\"DOI\"], \"\\n\\n\")\n\n{'cloud-info': {'Region': 'us-west-2',\n 'S3BucketAndObjectPrefixNames': ['podaac-ops-cumulus-public/SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205/',\n 'podaac-ops-cumulus-protected/SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205/'],\n 'S3CredentialsAPIDocumentationURL': 'https://archive.podaac.earthdata.nasa.gov/s3credentialsREADME',\n 'S3CredentialsAPIEndpoint': 'https://archive.podaac.earthdata.nasa.gov/s3credentials'},\n 'concept-id': 'C2270392799-POCLOUD',\n 'file-type': \"[{'Format': 'netCDF-4', 'FormatType': 'Native', \"\n \"'AverageFileSize': 9.7, 'AverageFileSizeUnit': 'MB'}]\",\n 'get-data': ['https://cmr.earthdata.nasa.gov/virtual-directory/collections/C2270392799-POCLOUD',\n 'https://search.earthdata.nasa.gov/search/granules?p=C2270392799-POCLOUD'],\n 'short-name': 'SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205',\n 'version': '2205'}\nNone This dataset provides gridded Sea Surface Height Anomalies (SSHA) above a mean sea surface, on a 1/6th degree grid every 5 days. It contains the fully corrected heights, with a delay of up to 3 months. The gridded data are derived from the along-track SSHA data of TOPEX/Poseidon, Jason-1, Jason-2, Jason-3 and Jason-CS (Sentinel-6) as reference data from the level 2 along-track data found at https://podaac.jpl.nasa.gov/dataset/MERGED_TP_J1_OSTM_OST_CYCLES_V51, plus ERS-1, ERS-2, Envisat, SARAL-AltiKa, CryoSat-2, Sentinel-3A, Sentinel-3B, depending on the date, from the RADS database. The date given in the grid files is the center of the 5-day window. The grids were produced from altimeter data using Kriging interpolation, which gives best linear prediction based upon prior knowledge of covariance. \n {'DOI': '10.5067/SLREF-CDRV3', 'Authority': 'https://doi.org'} \n\n\n{'cloud-info': {'Region': 'us-west-2',\n 'S3BucketAndObjectPrefixNames': ['nsidc-cumulus-prod-protected/ATLAS/ATL21/003',\n 'nsidc-cumulus-prod-public/ATLAS/ATL21/003'],\n 'S3CredentialsAPIDocumentationURL': 'https://data.nsidc.earthdatacloud.nasa.gov/s3credentialsREADME',\n 'S3CredentialsAPIEndpoint': 'https://data.nsidc.earthdatacloud.nasa.gov/s3credentials'},\n 'concept-id': 'C2753316241-NSIDC_CPRD',\n 'file-type': \"[{'FormatType': 'Native', 'Format': 'HDF5', \"\n \"'FormatDescription': 'HTTPS'}]\",\n 'get-data': ['https://search.earthdata.nasa.gov/search?q=ATL21+V003'],\n 'short-name': 'ATL21',\n 'version': '003'}\nNone This data set contains daily and monthly gridded polar sea surface height (SSH) anomalies, derived from the along-track ATLAS/ICESat-2 L3A Sea Ice Height product (ATL10, V6). The ATL10 product identifies leads in sea ice and establishes a reference sea surface used to estimate SSH in 10 km along-track segments. ATL21 aggregates the ATL10 along-track SSH estimates and computes daily and monthly gridded SSH anomaly in NSIDC Polar Stereographic Northern and Southern Hemisphere 25 km grids. \n {'DOI': '10.5067/ATLAS/ATL21.003'}" + "objectID": "in-development/CMR-STAC-Search.html#b.-search-for-items", + "href": "in-development/CMR-STAC-Search.html#b.-search-for-items", + "title": "Getting Started with NASA’s CMR-STAC API in R", + "section": "3b. Search for Items", + "text": "3b. Search for Items\nNext, submit a query to STAC Search endpoint using a POST request.\n\nsearch_req <- httr::POST(lpcloud_search_URL[1], body = body, encode = \"json\") %>% \n httr::content(as = \"text\") %>% \n jsonlite::fromJSON()\n\nnames(search_req)\n\n[1] \"type\" \"stac_version\" \"numberMatched\" \"numberReturned\"\n[5] \"features\" \"links\" \"context\" \n\n\nLet’s see how many STAC Items, or granules, intersect with our search parameters.\n\ncat(\"The number of STAC Items matched your query is \", search_req$numberMatched, 'and ', search_req$numberReturned, 'Items are returned.')\n\nThe number of STAC Items matched your query is 1 and 1 Items are returned.\n\n\nNext, create a data frame with the returned information, including granule ID, datetime properties, and the downloadable URL links to the assets.\n\ngranule_list <- list()\n\nn <- 1\nfor(row in row.names(search_req$features)){\n f <- search_req$features[row,]\n for (b in f$assets){\n df <- data.frame(Collection = f$collection,\n Granule_ID = f$id,\n Datetime = f$properties$datetime,\n Asset_Link = b$href, stringsAsFactors=FALSE)\n granule_list[[n]] <- df\n n <- n + 1\n }\n}\n\nsearch_df <- do.call(rbind, granule_list)\nDT::datatable(search_df)\n\n\n\n\n\n\nThe CMR-STAC Search endpoint allows user to quickly search for STAC Items that meet their specific spatial, temporal, and data product requirements. Now that you learned how to navigate and explore the CMR-STAC catalog, check out the HLS_tutorial to learn how to interact with HLS data specifically.\n\n\nContact Information\nMaterial written by Mahsa Jami1 and Aaron Friesz1\nContact: LPDAAC@usgs.gov\nVoice: +1-866-573-3222\nOrganization: Land Processes Distributed Active Archive Center (LP DAAC)\nWebsite: https://lpdaac.usgs.gov/ \nDate last modified: 09-13-2021\n1 KBR, Inc., contractor to the U.S. Geological Survey, Earth Resources Observation and Science (EROS) Center, Sioux Falls, South Dakota, USA. Work performed under USGS contract G0121D0001 for LP DAAC2.\n2 LP DAAC Work performed under NASA contract NNG14HH33I." }, { - "objectID": "tutorials/Sea_Level_Rise.html#access-data", - "href": "tutorials/Sea_Level_Rise.html#access-data", - "title": "Analyzing Sea Level Rise with NASA Earthdata", - "section": "Access Data", - "text": "Access Data\nThe first dataset looks like it contains data from many altimetry missions, let’s explore a bit more! We will access the first granule of the SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205 collection in the month of May for every year that data is available and open the granules via xarray.\n\ngranules = []\nfor year in range(1990, 2019):\n granule = earthaccess.granule_query().short_name(\"SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205\").temporal(f\"{year}-05\", f\"{year}-06\").get(1)\n if len(granule)>0:\n granules.append(granule[0])\nprint(f\"Total granules: {len(granules)}\") \n\nTotal granules: 26\n\n\n\n%%time\n\nds_SSH = xr.open_mfdataset(earthaccess.open(granules), chunks={})\nds_SSH\n\n Opening 26 granules, approx size: 0.0 GB\nCPU times: user 5.79 s, sys: 846 ms, total: 6.64 s\nWall time: 24.9 s\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (Time: 26, Longitude: 2160, nv: 2, Latitude: 960)\nCoordinates:\n * Longitude (Longitude) float32 0.08333 0.25 0.4167 ... 359.6 359.8 359.9\n * Latitude (Latitude) float32 -79.92 -79.75 -79.58 ... 79.58 79.75 79.92\n * Time (Time) datetime64[ns] 1993-05-03T12:00:00 ... 2018-05-02T12:...\nDimensions without coordinates: nv\nData variables:\n Lon_bounds (Time, Longitude, nv) float32 dask.array<chunksize=(1, 2160, 2), meta=np.ndarray>\n Lat_bounds (Time, Latitude, nv) float32 dask.array<chunksize=(1, 960, 2), meta=np.ndarray>\n Time_bounds (Time, nv) datetime64[ns] dask.array<chunksize=(1, 2), meta=np.ndarray>\n SLA (Time, Latitude, Longitude) float32 dask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n SLA_ERR (Time, Latitude, Longitude) float32 dask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\nAttributes: (12/21)\n Conventions: CF-1.6\n ncei_template_version: NCEI_NetCDF_Grid_Template_v2.0\n Institution: Jet Propulsion Laboratory\n geospatial_lat_min: -79.916664\n geospatial_lat_max: 79.916664\n geospatial_lon_min: 0.083333336\n ... ...\n version_number: 2205\n Data_Pnts_Each_Sat: {\"16\": 780190, \"1001\": 668288}\n source_version: commit dc95db885c920084614a41849ce5a7d417198ef3\n SLA_Global_MEAN: -0.027657876081274502\n SLA_Global_STD: 0.0859016072308609\n latency: finalxarray.DatasetDimensions:Time: 26Longitude: 2160nv: 2Latitude: 960Coordinates: (3)Longitude(Longitude)float320.08333 0.25 0.4167 ... 359.8 359.9standard_name :longitudeunits :degrees_eastpoint_spacing :evenlong_name :longitudeaxis :Xbounds :Lon_boundsarray([8.333334e-02, 2.500000e-01, 4.166667e-01, ..., 3.595833e+02,\n 3.597500e+02, 3.599167e+02], dtype=float32)Latitude(Latitude)float32-79.92 -79.75 ... 79.75 79.92standard_name :latitudeunits :degrees_northpoint_spacing :evenlong_name :latitudeaxis :Ybounds :Lat_boundsarray([-79.916664, -79.75 , -79.583336, ..., 79.583336, 79.75 ,\n 79.916664], dtype=float32)Time(Time)datetime64[ns]1993-05-03T12:00:00 ... 2018-05-...standard_name :timelong_name :Timebounds :Time_boundsaxis :Tarray(['1993-05-03T12:00:00.000000000', '1994-05-03T12:00:00.000000000',\n '1995-05-03T12:00:00.000000000', '1996-05-02T12:00:00.000000000',\n '1997-05-02T12:00:00.000000000', '1998-05-02T12:00:00.000000000',\n '1999-05-02T12:00:00.000000000', '2000-05-01T12:00:00.000000000',\n '2001-05-01T12:00:00.000000000', '2002-05-01T12:00:00.000000000',\n '2003-05-01T12:00:00.000000000', '2004-05-05T12:00:00.000000000',\n '2005-05-05T12:00:00.000000000', '2006-05-05T12:00:00.000000000',\n '2007-05-05T12:00:00.000000000', '2008-05-04T12:00:00.000000000',\n '2009-05-04T12:00:00.000000000', '2010-05-04T12:00:00.000000000',\n '2011-05-04T12:00:00.000000000', '2012-05-03T12:00:00.000000000',\n '2013-05-03T12:00:00.000000000', '2014-05-03T12:00:00.000000000',\n '2015-05-03T12:00:00.000000000', '2016-05-02T12:00:00.000000000',\n '2017-05-02T12:00:00.000000000', '2018-05-02T12:00:00.000000000'],\n dtype='datetime64[ns]')Data variables: (5)Lon_bounds(Time, Longitude, nv)float32dask.array<chunksize=(1, 2160, 2), meta=np.ndarray>units :degrees_eastcomment :longitude values at the west and east bounds of each pixel.\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n438.75 kiB\n16.88 kiB\n\n\nShape\n(26, 2160, 2)\n(1, 2160, 2)\n\n\nDask graph\n26 chunks in 79 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\nLat_bounds\n\n\n(Time, Latitude, nv)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2), meta=np.ndarray>\n\n\n\n\nunits :\n\ndegrees_north\n\ncomment :\n\nlatitude values at the north and south bounds of each pixel.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n195.00 kiB\n7.50 kiB\n\n\nShape\n(26, 960, 2)\n(1, 960, 2)\n\n\nDask graph\n26 chunks in 79 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nTime_bounds\n\n\n(Time, nv)\n\n\ndatetime64[ns]\n\n\ndask.array<chunksize=(1, 2), meta=np.ndarray>\n\n\n\n\ncomment :\n\nTime bounds for each time value, same value as time variable. The time variable is defined on points instead of on bounding boxes.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n416 B\n16 B\n\n\nShape\n(26, 2)\n(1, 2)\n\n\nDask graph\n26 chunks in 53 graph layers\n\n\nData type\ndatetime64[ns] numpy.ndarray\n\n\n\n\n\n\n\n\n\nSLA\n\n\n(Time, Latitude, Longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n\n\n\n\nunits :\n\nm\n\nlong_name :\n\nSea Level Anomaly Estimate\n\nstandard_name :\n\nsea_surface_height_above_sea_level\n\nalias :\n\nsea_surface_height_above_sea_level\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n205.66 MiB\n7.91 MiB\n\n\nShape\n(26, 960, 2160)\n(1, 960, 2160)\n\n\nDask graph\n26 chunks in 53 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nSLA_ERR\n\n\n(Time, Latitude, Longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n\n\n\n\nunits :\n\nm\n\nlong_name :\n\nSea Level Anomaly Error Estimate\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n205.66 MiB\n7.91 MiB\n\n\nShape\n(26, 960, 2160)\n(1, 960, 2160)\n\n\nDask graph\n26 chunks in 53 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\nIndexes: (3)LongitudePandasIndexPandasIndex(Index([0.0833333358168602, 0.25, 0.4166666567325592,\n 0.5833333134651184, 0.75, 0.9166666865348816,\n 1.0833333730697632, 1.25, 1.4166666269302368,\n 1.5833333730697632,\n ...\n 358.4166564941406, 358.5833435058594, 358.75,\n 358.9166564941406, 359.0833435058594, 359.25,\n 359.4166564941406, 359.5833435058594, 359.75,\n 359.9166564941406],\n dtype='float32', name='Longitude', length=2160))LatitudePandasIndexPandasIndex(Index([-79.91666412353516, -79.75, -79.58333587646484,\n -79.41666412353516, -79.25, -79.08333587646484,\n -78.91666412353516, -78.75, -78.58333587646484,\n -78.41666412353516,\n ...\n 78.41666412353516, 78.58333587646484, 78.75,\n 78.91666412353516, 79.08333587646484, 79.25,\n 79.41666412353516, 79.58333587646484, 79.75,\n 79.91666412353516],\n dtype='float32', name='Latitude', length=960))TimePandasIndexPandasIndex(DatetimeIndex(['1993-05-03 12:00:00', '1994-05-03 12:00:00',\n '1995-05-03 12:00:00', '1996-05-02 12:00:00',\n '1997-05-02 12:00:00', '1998-05-02 12:00:00',\n '1999-05-02 12:00:00', '2000-05-01 12:00:00',\n '2001-05-01 12:00:00', '2002-05-01 12:00:00',\n '2003-05-01 12:00:00', '2004-05-05 12:00:00',\n '2005-05-05 12:00:00', '2006-05-05 12:00:00',\n '2007-05-05 12:00:00', '2008-05-04 12:00:00',\n '2009-05-04 12:00:00', '2010-05-04 12:00:00',\n '2011-05-04 12:00:00', '2012-05-03 12:00:00',\n '2013-05-03 12:00:00', '2014-05-03 12:00:00',\n '2015-05-03 12:00:00', '2016-05-02 12:00:00',\n '2017-05-02 12:00:00', '2018-05-02 12:00:00'],\n dtype='datetime64[ns]', name='Time', freq=None))Attributes: (21)Conventions :CF-1.6ncei_template_version :NCEI_NetCDF_Grid_Template_v2.0Institution :Jet Propulsion Laboratorygeospatial_lat_min :-79.916664geospatial_lat_max :79.916664geospatial_lon_min :0.083333336geospatial_lon_max :359.91666time_coverage_start :1993-05-03time_coverage_end :1993-05-03date_created :2022-10-30T20:40:50.716092title :Sea Level Anomaly Estimate based on Altimeter Data, final product (replaced interim version).short_name :SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205long_name :MEaSUREs Gridded Sea Surface Height Anomalies Version 2205summary :Sea level anomaly grids from altimeter data using Kriging interpolation, which gives best linear prediction based upon prior knowledge of covariance. DOI :10.5067/SLREF-CDRV3version_number :2205Data_Pnts_Each_Sat :{\"16\": 780190, \"1001\": 668288}source_version :commit dc95db885c920084614a41849ce5a7d417198ef3SLA_Global_MEAN :-0.027657876081274502SLA_Global_STD :0.0859016072308609latency :final" + "objectID": "how-tos/using-s3-storage.html", + "href": "how-tos/using-s3-storage.html", + "title": "Using S3 Bucket Storage in NASA-Openscapes Hub", + "section": "", + "text": "When you are working in the NASA Openscapes Hub, there are strategies we can use to manage our storage both in terms of cost and performance. The default storage location is the HOME directory (/home/jovyan/) mounted to the compute instance (the cloud computer that is doing the computations). The Hub uses an EC2 compute instance, with the HOME directory mounted to AWS Elastic File System (EFS) storage. This drive is really handy because it is persistent across server restarts and is a great place to store your code. However the HOME directory is not a great place to store data, as it is very expensive, and can also be quite slow to read from and write to.\nTo that end, the hub provides every user access to two AWS S3 buckets - a “scratch” bucket for short-term storage, and a “persistent” bucket for longer-term storage. S3 buckets have fast read/write, and storage costs are relatively inexpensive compared to storing in your HOME directory. A useful way to think of S3 buckets in relation to your compute instance is like attaching a cheap but fast external hard drive to your expensive laptop.\nOne other thing to note about these buckets is that all hub users can access each other’s user directories. These buckets are accessible only when you are working inside the hub; you can access them using the environment variables:\n\n$SCRATCH_BUCKET pointing to s3://openscapeshub-scratch/[your-username]\n\nScratch buckets are designed for storage of temporary files, e.g. intermediate results. Objects stored in a scratch bucket are removed after 7 days from their creation.\n\n$PERSISTENT_BUCKET pointing to s3://openscapeshub-persistent/[your-username]\n\nPersistent buckets are designed for storing data that is consistently used throughout the lifetime of a project. There is no automatic purging of objects in persistent buckets, so it is the responsibility of the hub admin and/or hub users to delete objects when they are no longer needed to minimize cloud billing costs.\n\n\nWe can interact with these directories in Python using the packages boto3 and/or s3fs, or in a terminal with the awsv2 cli tool. This tutorial will focus on using the s3fs package. See this page for more information on using S3 buckets in a 2i2c hub, and tips on using the aws cli tool.\n\n\nWe will start by accessing the same data we did in the Earthdata Cloud Clinic - reading it into memory as an xarray object and subsetting it.\n\nimport earthaccess \nimport xarray as xr\nimport hvplot.xarray #plot\nimport os\nimport tempfile\nimport s3fs # aws s3 access\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\nauth = earthaccess.login()\n\n\ndata_name = \"SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205\"\n\nresults = earthaccess.search_data(\n short_name=data_name,\n cloud_hosted=True,\n temporal=(\"2021-07-01\", \"2021-09-30\"),\n)\n\nGranules found: 18\n\n\n\nds = xr.open_mfdataset(earthaccess.open(results))\nds\n\nOpening 18 granules, approx size: 0.16 GB\nusing endpoint: https://archive.podaac.earthdata.nasa.gov/s3credentials\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset> Size: 299MB\nDimensions: (Time: 18, Longitude: 2160, nv: 2, Latitude: 960)\nCoordinates:\n * Longitude (Longitude) float32 9kB 0.08333 0.25 0.4167 ... 359.8 359.9\n * Latitude (Latitude) float32 4kB -79.92 -79.75 -79.58 ... 79.75 79.92\n * Time (Time) datetime64[ns] 144B 2021-07-05T12:00:00 ... 2021-09-2...\nDimensions without coordinates: nv\nData variables:\n Lon_bounds (Time, Longitude, nv) float32 311kB dask.array<chunksize=(1, 2160, 2), meta=np.ndarray>\n Lat_bounds (Time, Latitude, nv) float32 138kB dask.array<chunksize=(1, 960, 2), meta=np.ndarray>\n Time_bounds (Time, nv) datetime64[ns] 288B dask.array<chunksize=(1, 2), meta=np.ndarray>\n SLA (Time, Latitude, Longitude) float32 149MB dask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n SLA_ERR (Time, Latitude, Longitude) float32 149MB dask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\nAttributes: (12/21)\n Conventions: CF-1.6\n ncei_template_version: NCEI_NetCDF_Grid_Template_v2.0\n Institution: Jet Propulsion Laboratory\n geospatial_lat_min: -79.916664\n geospatial_lat_max: 79.916664\n geospatial_lon_min: 0.083333336\n ... ...\n version_number: 2205\n Data_Pnts_Each_Sat: {\"16\": 743215, \"1007\": 674076}\n source_version: commit 58c7da13c0c0069ae940c33a82bf1544b7d991bf\n SLA_Global_MEAN: 0.06428374482174487\n SLA_Global_STD: 0.0905195660534004\n latency: finalxarray.DatasetDimensions:Time: 18Longitude: 2160nv: 2Latitude: 960Coordinates: (3)Longitude(Longitude)float320.08333 0.25 0.4167 ... 359.8 359.9standard_name :longitudeunits :degrees_eastpoint_spacing :evenlong_name :longitudeaxis :Xbounds :Lon_boundsarray([8.333334e-02, 2.500000e-01, 4.166667e-01, ..., 3.595833e+02,\n 3.597500e+02, 3.599167e+02], dtype=float32)Latitude(Latitude)float32-79.92 -79.75 ... 79.75 79.92standard_name :latitudeunits :degrees_northpoint_spacing :evenlong_name :latitudeaxis :Ybounds :Lat_boundsarray([-79.916664, -79.75 , -79.583336, ..., 79.583336, 79.75 ,\n 79.916664], dtype=float32)Time(Time)datetime64[ns]2021-07-05T12:00:00 ... 2021-09-...standard_name :timelong_name :Timebounds :Time_boundsaxis :Tarray(['2021-07-05T12:00:00.000000000', '2021-07-10T12:00:00.000000000',\n '2021-07-15T12:00:00.000000000', '2021-07-20T12:00:00.000000000',\n '2021-07-25T12:00:00.000000000', '2021-07-30T12:00:00.000000000',\n '2021-08-04T12:00:00.000000000', '2021-08-09T12:00:00.000000000',\n '2021-08-14T12:00:00.000000000', '2021-08-19T12:00:00.000000000',\n '2021-08-24T12:00:00.000000000', '2021-08-29T12:00:00.000000000',\n '2021-09-03T12:00:00.000000000', '2021-09-08T12:00:00.000000000',\n '2021-09-13T12:00:00.000000000', '2021-09-18T12:00:00.000000000',\n '2021-09-23T12:00:00.000000000', '2021-09-28T12:00:00.000000000'],\n dtype='datetime64[ns]')Data variables: (5)Lon_bounds(Time, Longitude, nv)float32dask.array<chunksize=(1, 2160, 2), meta=np.ndarray>units :degrees_eastcomment :longitude values at the west and east bounds of each pixel.\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n303.75 kiB\n16.88 kiB\n\n\nShape\n(18, 2160, 2)\n(1, 2160, 2)\n\n\nDask graph\n18 chunks in 55 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\nLat_bounds\n\n\n(Time, Latitude, nv)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2), meta=np.ndarray>\n\n\n\n\nunits :\n\ndegrees_north\n\ncomment :\n\nlatitude values at the north and south bounds of each pixel.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n135.00 kiB\n7.50 kiB\n\n\nShape\n(18, 960, 2)\n(1, 960, 2)\n\n\nDask graph\n18 chunks in 55 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nTime_bounds\n\n\n(Time, nv)\n\n\ndatetime64[ns]\n\n\ndask.array<chunksize=(1, 2), meta=np.ndarray>\n\n\n\n\ncomment :\n\nTime bounds for each time value, same value as time variable. The time variable is defined on points instead of on bounding boxes.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n288 B\n16 B\n\n\nShape\n(18, 2)\n(1, 2)\n\n\nDask graph\n18 chunks in 37 graph layers\n\n\nData type\ndatetime64[ns] numpy.ndarray\n\n\n\n\n\n\n\n\n\nSLA\n\n\n(Time, Latitude, Longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n\n\n\n\nunits :\n\nm\n\nlong_name :\n\nSea Level Anomaly Estimate\n\nstandard_name :\n\nsea_surface_height_above_sea_level\n\nalias :\n\nsea_surface_height_above_sea_level\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n142.38 MiB\n7.91 MiB\n\n\nShape\n(18, 960, 2160)\n(1, 960, 2160)\n\n\nDask graph\n18 chunks in 37 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nSLA_ERR\n\n\n(Time, Latitude, Longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n\n\n\n\nunits :\n\nm\n\nlong_name :\n\nSea Level Anomaly Error Estimate\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n142.38 MiB\n7.91 MiB\n\n\nShape\n(18, 960, 2160)\n(1, 960, 2160)\n\n\nDask graph\n18 chunks in 37 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\nIndexes: (3)LongitudePandasIndexPandasIndex(Index([0.0833333358168602, 0.25, 0.4166666567325592,\n 0.5833333134651184, 0.75, 0.9166666865348816,\n 1.0833333730697632, 1.25, 1.4166666269302368,\n 1.5833333730697632,\n ...\n 358.4166564941406, 358.5833435058594, 358.75,\n 358.9166564941406, 359.0833435058594, 359.25,\n 359.4166564941406, 359.5833435058594, 359.75,\n 359.9166564941406],\n dtype='float32', name='Longitude', length=2160))LatitudePandasIndexPandasIndex(Index([-79.91666412353516, -79.75, -79.58333587646484,\n -79.41666412353516, -79.25, -79.08333587646484,\n -78.91666412353516, -78.75, -78.58333587646484,\n -78.41666412353516,\n ...\n 78.41666412353516, 78.58333587646484, 78.75,\n 78.91666412353516, 79.08333587646484, 79.25,\n 79.41666412353516, 79.58333587646484, 79.75,\n 79.91666412353516],\n dtype='float32', name='Latitude', length=960))TimePandasIndexPandasIndex(DatetimeIndex(['2021-07-05 12:00:00', '2021-07-10 12:00:00',\n '2021-07-15 12:00:00', '2021-07-20 12:00:00',\n '2021-07-25 12:00:00', '2021-07-30 12:00:00',\n '2021-08-04 12:00:00', '2021-08-09 12:00:00',\n '2021-08-14 12:00:00', '2021-08-19 12:00:00',\n '2021-08-24 12:00:00', '2021-08-29 12:00:00',\n '2021-09-03 12:00:00', '2021-09-08 12:00:00',\n '2021-09-13 12:00:00', '2021-09-18 12:00:00',\n '2021-09-23 12:00:00', '2021-09-28 12:00:00'],\n dtype='datetime64[ns]', name='Time', freq=None))Attributes: (21)Conventions :CF-1.6ncei_template_version :NCEI_NetCDF_Grid_Template_v2.0Institution :Jet Propulsion Laboratorygeospatial_lat_min :-79.916664geospatial_lat_max :79.916664geospatial_lon_min :0.083333336geospatial_lon_max :359.91666time_coverage_start :2021-07-05time_coverage_end :2021-07-05date_created :2022-10-30T20:58:16.297288title :Sea Level Anomaly Estimate based on Altimeter Data, final product (replaced interim version).short_name :SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205long_name :MEaSUREs Gridded Sea Surface Height Anomalies Version 2205summary :Sea level anomaly grids from altimeter data using Kriging interpolation, which gives best linear prediction based upon prior knowledge of covariance. DOI :10.5067/SLREF-CDRV3version_number :2205Data_Pnts_Each_Sat :{\"16\": 743215, \"1007\": 674076}source_version :commit 58c7da13c0c0069ae940c33a82bf1544b7d991bfSLA_Global_MEAN :0.06428374482174487SLA_Global_STD :0.0905195660534004latency :final\n\n\n\nds_subset = ds['SLA'].sel(Latitude=slice(15.8, 35.9), Longitude=slice(234.5,260.5)) \nds_subset\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray 'SLA' (Time: 18, Latitude: 120, Longitude: 156)> Size: 1MB\ndask.array<getitem, shape=(18, 120, 156), dtype=float32, chunksize=(1, 120, 156), chunktype=numpy.ndarray>\nCoordinates:\n * Longitude (Longitude) float32 624B 234.6 234.8 234.9 ... 260.1 260.2 260.4\n * Latitude (Latitude) float32 480B 15.92 16.08 16.25 ... 35.42 35.58 35.75\n * Time (Time) datetime64[ns] 144B 2021-07-05T12:00:00 ... 2021-09-28T...\nAttributes:\n units: m\n long_name: Sea Level Anomaly Estimate\n standard_name: sea_surface_height_above_sea_level\n alias: sea_surface_height_above_sea_levelxarray.DataArray'SLA'Time: 18Latitude: 120Longitude: 156dask.array<chunksize=(1, 120, 156), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n1.29 MiB\n73.12 kiB\n\n\nShape\n(18, 120, 156)\n(1, 120, 156)\n\n\nDask graph\n18 chunks in 38 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\nCoordinates: (3)Longitude(Longitude)float32234.6 234.8 234.9 ... 260.2 260.4standard_name :longitudeunits :degrees_eastpoint_spacing :evenlong_name :longitudeaxis :Xbounds :Lon_boundsarray([234.58333, 234.75 , 234.91667, 235.08333, 235.25 , 235.41667,\n 235.58333, 235.75 , 235.91667, 236.08333, 236.25 , 236.41667,\n 236.58333, 236.75 , 236.91667, 237.08333, 237.25 , 237.41667,\n 237.58333, 237.75 , 237.91667, 238.08333, 238.25 , 238.41667,\n 238.58333, 238.75 , 238.91667, 239.08333, 239.25 , 239.41667,\n 239.58333, 239.75 , 239.91667, 240.08333, 240.25 , 240.41667,\n 240.58333, 240.75 , 240.91667, 241.08333, 241.25 , 241.41667,\n 241.58333, 241.75 , 241.91667, 242.08333, 242.25 , 242.41667,\n 242.58333, 242.75 , 242.91667, 243.08333, 243.25 , 243.41667,\n 243.58333, 243.75 , 243.91667, 244.08333, 244.25 , 244.41667,\n 244.58333, 244.75 , 244.91667, 245.08333, 245.25 , 245.41667,\n 245.58333, 245.75 , 245.91667, 246.08333, 246.25 , 246.41667,\n 246.58333, 246.75 , 246.91667, 247.08333, 247.25 , 247.41667,\n 247.58333, 247.75 , 247.91667, 248.08333, 248.25 , 248.41667,\n 248.58333, 248.75 , 248.91667, 249.08333, 249.25 , 249.41667,\n 249.58333, 249.75 , 249.91667, 250.08333, 250.25 , 250.41667,\n 250.58333, 250.75 , 250.91667, 251.08333, 251.25 , 251.41667,\n 251.58333, 251.75 , 251.91667, 252.08333, 252.25 , 252.41667,\n 252.58333, 252.75 , 252.91667, 253.08333, 253.25 , 253.41667,\n 253.58333, 253.75 , 253.91667, 254.08333, 254.25 , 254.41667,\n 254.58333, 254.75 , 254.91667, 255.08333, 255.25 , 255.41667,\n 255.58333, 255.75 , 255.91667, 256.08334, 256.25 , 256.41666,\n 256.58334, 256.75 , 256.91666, 257.08334, 257.25 , 257.41666,\n 257.58334, 257.75 , 257.91666, 258.08334, 258.25 , 258.41666,\n 258.58334, 258.75 , 258.91666, 259.08334, 259.25 , 259.41666,\n 259.58334, 259.75 , 259.91666, 260.08334, 260.25 , 260.41666],\n dtype=float32)Latitude(Latitude)float3215.92 16.08 16.25 ... 35.58 35.75standard_name :latitudeunits :degrees_northpoint_spacing :evenlong_name :latitudeaxis :Ybounds :Lat_boundsarray([15.916667, 16.083334, 16.25 , 16.416666, 16.583334, 16.75 ,\n 16.916666, 17.083334, 17.25 , 17.416666, 17.583334, 17.75 ,\n 17.916666, 18.083334, 18.25 , 18.416666, 18.583334, 18.75 ,\n 18.916666, 19.083334, 19.25 , 19.416666, 19.583334, 19.75 ,\n 19.916666, 20.083334, 20.25 , 20.416666, 20.583334, 20.75 ,\n 20.916666, 21.083334, 21.25 , 21.416666, 21.583334, 21.75 ,\n 21.916666, 22.083334, 22.25 , 22.416666, 22.583334, 22.75 ,\n 22.916666, 23.083334, 23.25 , 23.416666, 23.583334, 23.75 ,\n 23.916666, 24.083334, 24.25 , 24.416666, 24.583334, 24.75 ,\n 24.916666, 25.083334, 25.25 , 25.416666, 25.583334, 25.75 ,\n 25.916666, 26.083334, 26.25 , 26.416666, 26.583334, 26.75 ,\n 26.916666, 27.083334, 27.25 , 27.416666, 27.583334, 27.75 ,\n 27.916666, 28.083334, 28.25 , 28.416666, 28.583334, 28.75 ,\n 28.916666, 29.083334, 29.25 , 29.416666, 29.583334, 29.75 ,\n 29.916666, 30.083334, 30.25 , 30.416666, 30.583334, 30.75 ,\n 30.916666, 31.083334, 31.25 , 31.416666, 31.583334, 31.75 ,\n 31.916666, 32.083332, 32.25 , 32.416668, 32.583332, 32.75 ,\n 32.916668, 33.083332, 33.25 , 33.416668, 33.583332, 33.75 ,\n 33.916668, 34.083332, 34.25 , 34.416668, 34.583332, 34.75 ,\n 34.916668, 35.083332, 35.25 , 35.416668, 35.583332, 35.75 ],\n dtype=float32)Time(Time)datetime64[ns]2021-07-05T12:00:00 ... 2021-09-...standard_name :timelong_name :Timebounds :Time_boundsaxis :Tarray(['2021-07-05T12:00:00.000000000', '2021-07-10T12:00:00.000000000',\n '2021-07-15T12:00:00.000000000', '2021-07-20T12:00:00.000000000',\n '2021-07-25T12:00:00.000000000', '2021-07-30T12:00:00.000000000',\n '2021-08-04T12:00:00.000000000', '2021-08-09T12:00:00.000000000',\n '2021-08-14T12:00:00.000000000', '2021-08-19T12:00:00.000000000',\n '2021-08-24T12:00:00.000000000', '2021-08-29T12:00:00.000000000',\n '2021-09-03T12:00:00.000000000', '2021-09-08T12:00:00.000000000',\n '2021-09-13T12:00:00.000000000', '2021-09-18T12:00:00.000000000',\n '2021-09-23T12:00:00.000000000', '2021-09-28T12:00:00.000000000'],\n dtype='datetime64[ns]')Indexes: (3)LongitudePandasIndexPandasIndex(Index([234.5833282470703, 234.75, 234.9166717529297,\n 235.0833282470703, 235.25, 235.4166717529297,\n 235.5833282470703, 235.75, 235.9166717529297,\n 236.0833282470703,\n ...\n 258.9166564941406, 259.0833435058594, 259.25,\n 259.4166564941406, 259.5833435058594, 259.75,\n 259.9166564941406, 260.0833435058594, 260.25,\n 260.4166564941406],\n dtype='float32', name='Longitude', length=156))LatitudePandasIndexPandasIndex(Index([15.916666984558105, 16.08333396911621, 16.25,\n 16.41666603088379, 16.58333396911621, 16.75,\n 16.91666603088379, 17.08333396911621, 17.25,\n 17.41666603088379,\n ...\n 34.25, 34.41666793823242, 34.58333206176758,\n 34.75, 34.91666793823242, 35.08333206176758,\n 35.25, 35.41666793823242, 35.58333206176758,\n 35.75],\n dtype='float32', name='Latitude', length=120))TimePandasIndexPandasIndex(DatetimeIndex(['2021-07-05 12:00:00', '2021-07-10 12:00:00',\n '2021-07-15 12:00:00', '2021-07-20 12:00:00',\n '2021-07-25 12:00:00', '2021-07-30 12:00:00',\n '2021-08-04 12:00:00', '2021-08-09 12:00:00',\n '2021-08-14 12:00:00', '2021-08-19 12:00:00',\n '2021-08-24 12:00:00', '2021-08-29 12:00:00',\n '2021-09-03 12:00:00', '2021-09-08 12:00:00',\n '2021-09-13 12:00:00', '2021-09-18 12:00:00',\n '2021-09-23 12:00:00', '2021-09-28 12:00:00'],\n dtype='datetime64[ns]', name='Time', freq=None))Attributes: (4)units :mlong_name :Sea Level Anomaly Estimatestandard_name :sea_surface_height_above_sea_levelalias :sea_surface_height_above_sea_level\n\n\n\n\n\nImagining this ds_subset object is now an important intermediate dataset, or the result of a complex analysis and we want to save it. Our default action might be to just save it to our HOME directory. This is simple, but we want to avoid this as it incurs significant storage costs, and using this data later will be slow.\nds_subset.to_netcdf(\"test.nc\") # avoid writing to home directory like this\n\n\n\ns3fs is a Python library that allows us to interact with S3 objects in a file-system like manner.\n\n# Create a S3FileSystem class\ns3 = s3fs.S3FileSystem()\n\n# Get scratch and persistent buckets\nscratch = os.environ[\"SCRATCH_BUCKET\"]\npersistent = os.environ[\"PERSISTENT_BUCKET\"]\n\nprint(scratch)\nprint(persistent)\n\ns3://openscapeshub-scratch/ateucher\ns3://openscapeshub-persistent/ateucher\n\n\nOur user-specific directories in the two buckets aren’t actually created until we put something in them, so if we try to check for their existence or list their contents before they are created, we will get an error. We will use the S3FileSystem.touch() method to place a simple empty file called .placeholder in each one to bring them into existence.\n\ns3.touch(f\"{scratch}/.placeholder\")\n\ns3.ls(scratch)\n\n['openscapeshub-scratch/ateucher/.placeholder']\n\n\nand in our persistent bucket:\n\ns3.touch(f\"{persistent}/.placeholder\")\n\ns3.ls(persistent)\n\n['openscapeshub-persistent/ateucher/.placeholder']\n\n\n(Note that adding these placeholders isn’t strictly necessary, as the first time you write anything to these buckets they will be created.)\n\n\n\nNext we can save ds_subset as a netcdf file in our scratch bucket. This involves writing to a temporary directory first, and then moving that to the SCRATCH bucket:\n\n# Where we want to store it:\nscratch_nc_file_path = f\"{scratch}/test123.nc\"\n\n# Create a temporary intermediate file and save it to the bucket\nwith tempfile.NamedTemporaryFile(suffix = \".nc\") as tmp:\n ds_subset.to_netcdf(tmp.name) # save it to a temporary file\n s3.put(tmp.name, scratch_nc_file_path) # move that file to the scratch bucket\n\n# Ensure the file is there\ns3.ls(scratch)\n\n['openscapeshub-scratch/ateucher/.placeholder',\n 'openscapeshub-scratch/ateucher/test123.nc']\n\n\nAnd we can open it to ensure it worked:\n\nds_from_scratch = xr.open_dataarray(s3.open(scratch_nc_file_path))\n\nds_from_scratch\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray 'SLA' (Time: 18, Latitude: 120, Longitude: 156)> Size: 1MB\n[336960 values with dtype=float32]\nCoordinates:\n * Longitude (Longitude) float32 624B 234.6 234.8 234.9 ... 260.1 260.2 260.4\n * Latitude (Latitude) float32 480B 15.92 16.08 16.25 ... 35.42 35.58 35.75\n * Time (Time) datetime64[ns] 144B 2021-07-05T12:00:00 ... 2021-09-28T...\nAttributes:\n units: m\n long_name: Sea Level Anomaly Estimate\n standard_name: sea_surface_height_above_sea_level\n alias: sea_surface_height_above_sea_levelxarray.DataArray'SLA'Time: 18Latitude: 120Longitude: 156...[336960 values with dtype=float32]Coordinates: (3)Longitude(Longitude)float32234.6 234.8 234.9 ... 260.2 260.4standard_name :longitudeunits :degrees_eastpoint_spacing :evenlong_name :longitudeaxis :Xbounds :Lon_boundsarray([234.58333, 234.75 , 234.91667, 235.08333, 235.25 , 235.41667,\n 235.58333, 235.75 , 235.91667, 236.08333, 236.25 , 236.41667,\n 236.58333, 236.75 , 236.91667, 237.08333, 237.25 , 237.41667,\n 237.58333, 237.75 , 237.91667, 238.08333, 238.25 , 238.41667,\n 238.58333, 238.75 , 238.91667, 239.08333, 239.25 , 239.41667,\n 239.58333, 239.75 , 239.91667, 240.08333, 240.25 , 240.41667,\n 240.58333, 240.75 , 240.91667, 241.08333, 241.25 , 241.41667,\n 241.58333, 241.75 , 241.91667, 242.08333, 242.25 , 242.41667,\n 242.58333, 242.75 , 242.91667, 243.08333, 243.25 , 243.41667,\n 243.58333, 243.75 , 243.91667, 244.08333, 244.25 , 244.41667,\n 244.58333, 244.75 , 244.91667, 245.08333, 245.25 , 245.41667,\n 245.58333, 245.75 , 245.91667, 246.08333, 246.25 , 246.41667,\n 246.58333, 246.75 , 246.91667, 247.08333, 247.25 , 247.41667,\n 247.58333, 247.75 , 247.91667, 248.08333, 248.25 , 248.41667,\n 248.58333, 248.75 , 248.91667, 249.08333, 249.25 , 249.41667,\n 249.58333, 249.75 , 249.91667, 250.08333, 250.25 , 250.41667,\n 250.58333, 250.75 , 250.91667, 251.08333, 251.25 , 251.41667,\n 251.58333, 251.75 , 251.91667, 252.08333, 252.25 , 252.41667,\n 252.58333, 252.75 , 252.91667, 253.08333, 253.25 , 253.41667,\n 253.58333, 253.75 , 253.91667, 254.08333, 254.25 , 254.41667,\n 254.58333, 254.75 , 254.91667, 255.08333, 255.25 , 255.41667,\n 255.58333, 255.75 , 255.91667, 256.08334, 256.25 , 256.41666,\n 256.58334, 256.75 , 256.91666, 257.08334, 257.25 , 257.41666,\n 257.58334, 257.75 , 257.91666, 258.08334, 258.25 , 258.41666,\n 258.58334, 258.75 , 258.91666, 259.08334, 259.25 , 259.41666,\n 259.58334, 259.75 , 259.91666, 260.08334, 260.25 , 260.41666],\n dtype=float32)Latitude(Latitude)float3215.92 16.08 16.25 ... 35.58 35.75standard_name :latitudeunits :degrees_northpoint_spacing :evenlong_name :latitudeaxis :Ybounds :Lat_boundsarray([15.916667, 16.083334, 16.25 , 16.416666, 16.583334, 16.75 ,\n 16.916666, 17.083334, 17.25 , 17.416666, 17.583334, 17.75 ,\n 17.916666, 18.083334, 18.25 , 18.416666, 18.583334, 18.75 ,\n 18.916666, 19.083334, 19.25 , 19.416666, 19.583334, 19.75 ,\n 19.916666, 20.083334, 20.25 , 20.416666, 20.583334, 20.75 ,\n 20.916666, 21.083334, 21.25 , 21.416666, 21.583334, 21.75 ,\n 21.916666, 22.083334, 22.25 , 22.416666, 22.583334, 22.75 ,\n 22.916666, 23.083334, 23.25 , 23.416666, 23.583334, 23.75 ,\n 23.916666, 24.083334, 24.25 , 24.416666, 24.583334, 24.75 ,\n 24.916666, 25.083334, 25.25 , 25.416666, 25.583334, 25.75 ,\n 25.916666, 26.083334, 26.25 , 26.416666, 26.583334, 26.75 ,\n 26.916666, 27.083334, 27.25 , 27.416666, 27.583334, 27.75 ,\n 27.916666, 28.083334, 28.25 , 28.416666, 28.583334, 28.75 ,\n 28.916666, 29.083334, 29.25 , 29.416666, 29.583334, 29.75 ,\n 29.916666, 30.083334, 30.25 , 30.416666, 30.583334, 30.75 ,\n 30.916666, 31.083334, 31.25 , 31.416666, 31.583334, 31.75 ,\n 31.916666, 32.083332, 32.25 , 32.416668, 32.583332, 32.75 ,\n 32.916668, 33.083332, 33.25 , 33.416668, 33.583332, 33.75 ,\n 33.916668, 34.083332, 34.25 , 34.416668, 34.583332, 34.75 ,\n 34.916668, 35.083332, 35.25 , 35.416668, 35.583332, 35.75 ],\n dtype=float32)Time(Time)datetime64[ns]2021-07-05T12:00:00 ... 2021-09-...standard_name :timelong_name :Timebounds :Time_boundsaxis :Tarray(['2021-07-05T12:00:00.000000000', '2021-07-10T12:00:00.000000000',\n '2021-07-15T12:00:00.000000000', '2021-07-20T12:00:00.000000000',\n '2021-07-25T12:00:00.000000000', '2021-07-30T12:00:00.000000000',\n '2021-08-04T12:00:00.000000000', '2021-08-09T12:00:00.000000000',\n '2021-08-14T12:00:00.000000000', '2021-08-19T12:00:00.000000000',\n '2021-08-24T12:00:00.000000000', '2021-08-29T12:00:00.000000000',\n '2021-09-03T12:00:00.000000000', '2021-09-08T12:00:00.000000000',\n '2021-09-13T12:00:00.000000000', '2021-09-18T12:00:00.000000000',\n '2021-09-23T12:00:00.000000000', '2021-09-28T12:00:00.000000000'],\n dtype='datetime64[ns]')Indexes: (3)LongitudePandasIndexPandasIndex(Index([234.5833282470703, 234.75, 234.9166717529297,\n 235.0833282470703, 235.25, 235.4166717529297,\n 235.5833282470703, 235.75, 235.9166717529297,\n 236.0833282470703,\n ...\n 258.9166564941406, 259.0833435058594, 259.25,\n 259.4166564941406, 259.5833435058594, 259.75,\n 259.9166564941406, 260.0833435058594, 260.25,\n 260.4166564941406],\n dtype='float32', name='Longitude', length=156))LatitudePandasIndexPandasIndex(Index([15.916666984558105, 16.08333396911621, 16.25,\n 16.41666603088379, 16.58333396911621, 16.75,\n 16.91666603088379, 17.08333396911621, 17.25,\n 17.41666603088379,\n ...\n 34.25, 34.41666793823242, 34.58333206176758,\n 34.75, 34.91666793823242, 35.08333206176758,\n 35.25, 35.41666793823242, 35.58333206176758,\n 35.75],\n dtype='float32', name='Latitude', length=120))TimePandasIndexPandasIndex(DatetimeIndex(['2021-07-05 12:00:00', '2021-07-10 12:00:00',\n '2021-07-15 12:00:00', '2021-07-20 12:00:00',\n '2021-07-25 12:00:00', '2021-07-30 12:00:00',\n '2021-08-04 12:00:00', '2021-08-09 12:00:00',\n '2021-08-14 12:00:00', '2021-08-19 12:00:00',\n '2021-08-24 12:00:00', '2021-08-29 12:00:00',\n '2021-09-03 12:00:00', '2021-09-08 12:00:00',\n '2021-09-13 12:00:00', '2021-09-18 12:00:00',\n '2021-09-23 12:00:00', '2021-09-28 12:00:00'],\n dtype='datetime64[ns]', name='Time', freq=None))Attributes: (4)units :mlong_name :Sea Level Anomaly Estimatestandard_name :sea_surface_height_above_sea_levelalias :sea_surface_height_above_sea_level\n\n\n\nds_from_scratch.hvplot.image(x='Longitude', y='Latitude', cmap='RdBu', clim=(-0.5, 0.5), title=\"Sea Level Anomaly Estimate (m)\")\n\n\n\n\n\n \n\n\n\n\n\n\n\nIf we decide this is a file we want to keep around for a longer time period, we can move it to our persistent bucket. We can even make a subdirectory in our persistent bucket to keep us organized:\n\npersistent_dest_dir = f\"{persistent}/my-analysis-data/\"\n\n# Make directory in persistent bucket\ns3.mkdir(persistent_dest_dir)\n\n# Move the file\ns3.mv(scratch_nc_file_path, persistent_dest_dir)\n\n# Check the scratch and persistent bucket listings:\ns3.ls(scratch)\n\n['openscapeshub-scratch/ateucher/.placeholder']\n\n\n\ns3.ls(persistent)\n\n['openscapeshub-persistent/ateucher/.placeholder',\n 'openscapeshub-persistent/ateucher/my-analysis-data']\n\n\n\ns3.ls(persistent_dest_dir)\n\n['openscapeshub-persistent/ateucher/my-analysis-data/test123.nc']\n\n\n\n\n\nYou may already have some data in your HOME directory that you would like to move out to a persistent bucket. You can do that using the awsv2 s3 command line tool, which is already installed on the hub. You can open a terminal from the Hub Launcher - it will open in your HOME directory. You can then use the awsv2 s3 mv command to move a file to your bucket.\n\n\n$ awsv2 s3 mv my-big-file.nc $PERSISTENT_BUCKET/ # The trailing slash is important here\nmove: ./my-big-file.nc to s3://openscapeshub-persistent/ateucher/my-big-file.nc\n\n\n\nList the contents of the local results-data directory:\n$ ls results-data/\nmy-big-file1.nc my-big-file2.nc\nUse awsv2 s3 mv with the --recursive flag to move all files in a directory to a new directory in PERSISTENT_BUCKET\n$ awsv2 s3 mv --recursive results-data $PERSISTENT_BUCKET/results-data/\nmove: results-data/my-big-file1.nc to s3://openscapeshub-persistent/ateucher/results-data/my-big-file1.nc\nmove: results-data/my-big-file2.nc to s3://openscapeshub-persistent/ateucher/results-data/my-big-file2.nc" }, { - "objectID": "tutorials/Sea_Level_Rise.html#plot-sea-level-anomalies", - "href": "tutorials/Sea_Level_Rise.html#plot-sea-level-anomalies", - "title": "Analyzing Sea Level Rise with NASA Earthdata", - "section": "Plot Sea Level Anomalies", - "text": "Plot Sea Level Anomalies\n\nds_SSH.SLA.hvplot.image(y='Latitude', x='Longitude', cmap='viridis')" + "objectID": "how-tos/using-s3-storage.html#reading-and-writing-to-the-scratch_bucket", + "href": "how-tos/using-s3-storage.html#reading-and-writing-to-the-scratch_bucket", + "title": "Using S3 Bucket Storage in NASA-Openscapes Hub", + "section": "", + "text": "We will start by accessing the same data we did in the Earthdata Cloud Clinic - reading it into memory as an xarray object and subsetting it.\n\nimport earthaccess \nimport xarray as xr\nimport hvplot.xarray #plot\nimport os\nimport tempfile\nimport s3fs # aws s3 access\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\nauth = earthaccess.login()\n\n\ndata_name = \"SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205\"\n\nresults = earthaccess.search_data(\n short_name=data_name,\n cloud_hosted=True,\n temporal=(\"2021-07-01\", \"2021-09-30\"),\n)\n\nGranules found: 18\n\n\n\nds = xr.open_mfdataset(earthaccess.open(results))\nds\n\nOpening 18 granules, approx size: 0.16 GB\nusing endpoint: https://archive.podaac.earthdata.nasa.gov/s3credentials\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset> Size: 299MB\nDimensions: (Time: 18, Longitude: 2160, nv: 2, Latitude: 960)\nCoordinates:\n * Longitude (Longitude) float32 9kB 0.08333 0.25 0.4167 ... 359.8 359.9\n * Latitude (Latitude) float32 4kB -79.92 -79.75 -79.58 ... 79.75 79.92\n * Time (Time) datetime64[ns] 144B 2021-07-05T12:00:00 ... 2021-09-2...\nDimensions without coordinates: nv\nData variables:\n Lon_bounds (Time, Longitude, nv) float32 311kB dask.array<chunksize=(1, 2160, 2), meta=np.ndarray>\n Lat_bounds (Time, Latitude, nv) float32 138kB dask.array<chunksize=(1, 960, 2), meta=np.ndarray>\n Time_bounds (Time, nv) datetime64[ns] 288B dask.array<chunksize=(1, 2), meta=np.ndarray>\n SLA (Time, Latitude, Longitude) float32 149MB dask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n SLA_ERR (Time, Latitude, Longitude) float32 149MB dask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\nAttributes: (12/21)\n Conventions: CF-1.6\n ncei_template_version: NCEI_NetCDF_Grid_Template_v2.0\n Institution: Jet Propulsion Laboratory\n geospatial_lat_min: -79.916664\n geospatial_lat_max: 79.916664\n geospatial_lon_min: 0.083333336\n ... ...\n version_number: 2205\n Data_Pnts_Each_Sat: {\"16\": 743215, \"1007\": 674076}\n source_version: commit 58c7da13c0c0069ae940c33a82bf1544b7d991bf\n SLA_Global_MEAN: 0.06428374482174487\n SLA_Global_STD: 0.0905195660534004\n latency: finalxarray.DatasetDimensions:Time: 18Longitude: 2160nv: 2Latitude: 960Coordinates: (3)Longitude(Longitude)float320.08333 0.25 0.4167 ... 359.8 359.9standard_name :longitudeunits :degrees_eastpoint_spacing :evenlong_name :longitudeaxis :Xbounds :Lon_boundsarray([8.333334e-02, 2.500000e-01, 4.166667e-01, ..., 3.595833e+02,\n 3.597500e+02, 3.599167e+02], dtype=float32)Latitude(Latitude)float32-79.92 -79.75 ... 79.75 79.92standard_name :latitudeunits :degrees_northpoint_spacing :evenlong_name :latitudeaxis :Ybounds :Lat_boundsarray([-79.916664, -79.75 , -79.583336, ..., 79.583336, 79.75 ,\n 79.916664], dtype=float32)Time(Time)datetime64[ns]2021-07-05T12:00:00 ... 2021-09-...standard_name :timelong_name :Timebounds :Time_boundsaxis :Tarray(['2021-07-05T12:00:00.000000000', '2021-07-10T12:00:00.000000000',\n '2021-07-15T12:00:00.000000000', '2021-07-20T12:00:00.000000000',\n '2021-07-25T12:00:00.000000000', '2021-07-30T12:00:00.000000000',\n '2021-08-04T12:00:00.000000000', '2021-08-09T12:00:00.000000000',\n '2021-08-14T12:00:00.000000000', '2021-08-19T12:00:00.000000000',\n '2021-08-24T12:00:00.000000000', '2021-08-29T12:00:00.000000000',\n '2021-09-03T12:00:00.000000000', '2021-09-08T12:00:00.000000000',\n '2021-09-13T12:00:00.000000000', '2021-09-18T12:00:00.000000000',\n '2021-09-23T12:00:00.000000000', '2021-09-28T12:00:00.000000000'],\n dtype='datetime64[ns]')Data variables: (5)Lon_bounds(Time, Longitude, nv)float32dask.array<chunksize=(1, 2160, 2), meta=np.ndarray>units :degrees_eastcomment :longitude values at the west and east bounds of each pixel.\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n303.75 kiB\n16.88 kiB\n\n\nShape\n(18, 2160, 2)\n(1, 2160, 2)\n\n\nDask graph\n18 chunks in 55 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\nLat_bounds\n\n\n(Time, Latitude, nv)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2), meta=np.ndarray>\n\n\n\n\nunits :\n\ndegrees_north\n\ncomment :\n\nlatitude values at the north and south bounds of each pixel.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n135.00 kiB\n7.50 kiB\n\n\nShape\n(18, 960, 2)\n(1, 960, 2)\n\n\nDask graph\n18 chunks in 55 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nTime_bounds\n\n\n(Time, nv)\n\n\ndatetime64[ns]\n\n\ndask.array<chunksize=(1, 2), meta=np.ndarray>\n\n\n\n\ncomment :\n\nTime bounds for each time value, same value as time variable. The time variable is defined on points instead of on bounding boxes.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n288 B\n16 B\n\n\nShape\n(18, 2)\n(1, 2)\n\n\nDask graph\n18 chunks in 37 graph layers\n\n\nData type\ndatetime64[ns] numpy.ndarray\n\n\n\n\n\n\n\n\n\nSLA\n\n\n(Time, Latitude, Longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n\n\n\n\nunits :\n\nm\n\nlong_name :\n\nSea Level Anomaly Estimate\n\nstandard_name :\n\nsea_surface_height_above_sea_level\n\nalias :\n\nsea_surface_height_above_sea_level\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n142.38 MiB\n7.91 MiB\n\n\nShape\n(18, 960, 2160)\n(1, 960, 2160)\n\n\nDask graph\n18 chunks in 37 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nSLA_ERR\n\n\n(Time, Latitude, Longitude)\n\n\nfloat32\n\n\ndask.array<chunksize=(1, 960, 2160), meta=np.ndarray>\n\n\n\n\nunits :\n\nm\n\nlong_name :\n\nSea Level Anomaly Error Estimate\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n142.38 MiB\n7.91 MiB\n\n\nShape\n(18, 960, 2160)\n(1, 960, 2160)\n\n\nDask graph\n18 chunks in 37 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\nIndexes: (3)LongitudePandasIndexPandasIndex(Index([0.0833333358168602, 0.25, 0.4166666567325592,\n 0.5833333134651184, 0.75, 0.9166666865348816,\n 1.0833333730697632, 1.25, 1.4166666269302368,\n 1.5833333730697632,\n ...\n 358.4166564941406, 358.5833435058594, 358.75,\n 358.9166564941406, 359.0833435058594, 359.25,\n 359.4166564941406, 359.5833435058594, 359.75,\n 359.9166564941406],\n dtype='float32', name='Longitude', length=2160))LatitudePandasIndexPandasIndex(Index([-79.91666412353516, -79.75, -79.58333587646484,\n -79.41666412353516, -79.25, -79.08333587646484,\n -78.91666412353516, -78.75, -78.58333587646484,\n -78.41666412353516,\n ...\n 78.41666412353516, 78.58333587646484, 78.75,\n 78.91666412353516, 79.08333587646484, 79.25,\n 79.41666412353516, 79.58333587646484, 79.75,\n 79.91666412353516],\n dtype='float32', name='Latitude', length=960))TimePandasIndexPandasIndex(DatetimeIndex(['2021-07-05 12:00:00', '2021-07-10 12:00:00',\n '2021-07-15 12:00:00', '2021-07-20 12:00:00',\n '2021-07-25 12:00:00', '2021-07-30 12:00:00',\n '2021-08-04 12:00:00', '2021-08-09 12:00:00',\n '2021-08-14 12:00:00', '2021-08-19 12:00:00',\n '2021-08-24 12:00:00', '2021-08-29 12:00:00',\n '2021-09-03 12:00:00', '2021-09-08 12:00:00',\n '2021-09-13 12:00:00', '2021-09-18 12:00:00',\n '2021-09-23 12:00:00', '2021-09-28 12:00:00'],\n dtype='datetime64[ns]', name='Time', freq=None))Attributes: (21)Conventions :CF-1.6ncei_template_version :NCEI_NetCDF_Grid_Template_v2.0Institution :Jet Propulsion Laboratorygeospatial_lat_min :-79.916664geospatial_lat_max :79.916664geospatial_lon_min :0.083333336geospatial_lon_max :359.91666time_coverage_start :2021-07-05time_coverage_end :2021-07-05date_created :2022-10-30T20:58:16.297288title :Sea Level Anomaly Estimate based on Altimeter Data, final product (replaced interim version).short_name :SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205long_name :MEaSUREs Gridded Sea Surface Height Anomalies Version 2205summary :Sea level anomaly grids from altimeter data using Kriging interpolation, which gives best linear prediction based upon prior knowledge of covariance. DOI :10.5067/SLREF-CDRV3version_number :2205Data_Pnts_Each_Sat :{\"16\": 743215, \"1007\": 674076}source_version :commit 58c7da13c0c0069ae940c33a82bf1544b7d991bfSLA_Global_MEAN :0.06428374482174487SLA_Global_STD :0.0905195660534004latency :final\n\n\n\nds_subset = ds['SLA'].sel(Latitude=slice(15.8, 35.9), Longitude=slice(234.5,260.5)) \nds_subset\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray 'SLA' (Time: 18, Latitude: 120, Longitude: 156)> Size: 1MB\ndask.array<getitem, shape=(18, 120, 156), dtype=float32, chunksize=(1, 120, 156), chunktype=numpy.ndarray>\nCoordinates:\n * Longitude (Longitude) float32 624B 234.6 234.8 234.9 ... 260.1 260.2 260.4\n * Latitude (Latitude) float32 480B 15.92 16.08 16.25 ... 35.42 35.58 35.75\n * Time (Time) datetime64[ns] 144B 2021-07-05T12:00:00 ... 2021-09-28T...\nAttributes:\n units: m\n long_name: Sea Level Anomaly Estimate\n standard_name: sea_surface_height_above_sea_level\n alias: sea_surface_height_above_sea_levelxarray.DataArray'SLA'Time: 18Latitude: 120Longitude: 156dask.array<chunksize=(1, 120, 156), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n1.29 MiB\n73.12 kiB\n\n\nShape\n(18, 120, 156)\n(1, 120, 156)\n\n\nDask graph\n18 chunks in 38 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\nCoordinates: (3)Longitude(Longitude)float32234.6 234.8 234.9 ... 260.2 260.4standard_name :longitudeunits :degrees_eastpoint_spacing :evenlong_name :longitudeaxis :Xbounds :Lon_boundsarray([234.58333, 234.75 , 234.91667, 235.08333, 235.25 , 235.41667,\n 235.58333, 235.75 , 235.91667, 236.08333, 236.25 , 236.41667,\n 236.58333, 236.75 , 236.91667, 237.08333, 237.25 , 237.41667,\n 237.58333, 237.75 , 237.91667, 238.08333, 238.25 , 238.41667,\n 238.58333, 238.75 , 238.91667, 239.08333, 239.25 , 239.41667,\n 239.58333, 239.75 , 239.91667, 240.08333, 240.25 , 240.41667,\n 240.58333, 240.75 , 240.91667, 241.08333, 241.25 , 241.41667,\n 241.58333, 241.75 , 241.91667, 242.08333, 242.25 , 242.41667,\n 242.58333, 242.75 , 242.91667, 243.08333, 243.25 , 243.41667,\n 243.58333, 243.75 , 243.91667, 244.08333, 244.25 , 244.41667,\n 244.58333, 244.75 , 244.91667, 245.08333, 245.25 , 245.41667,\n 245.58333, 245.75 , 245.91667, 246.08333, 246.25 , 246.41667,\n 246.58333, 246.75 , 246.91667, 247.08333, 247.25 , 247.41667,\n 247.58333, 247.75 , 247.91667, 248.08333, 248.25 , 248.41667,\n 248.58333, 248.75 , 248.91667, 249.08333, 249.25 , 249.41667,\n 249.58333, 249.75 , 249.91667, 250.08333, 250.25 , 250.41667,\n 250.58333, 250.75 , 250.91667, 251.08333, 251.25 , 251.41667,\n 251.58333, 251.75 , 251.91667, 252.08333, 252.25 , 252.41667,\n 252.58333, 252.75 , 252.91667, 253.08333, 253.25 , 253.41667,\n 253.58333, 253.75 , 253.91667, 254.08333, 254.25 , 254.41667,\n 254.58333, 254.75 , 254.91667, 255.08333, 255.25 , 255.41667,\n 255.58333, 255.75 , 255.91667, 256.08334, 256.25 , 256.41666,\n 256.58334, 256.75 , 256.91666, 257.08334, 257.25 , 257.41666,\n 257.58334, 257.75 , 257.91666, 258.08334, 258.25 , 258.41666,\n 258.58334, 258.75 , 258.91666, 259.08334, 259.25 , 259.41666,\n 259.58334, 259.75 , 259.91666, 260.08334, 260.25 , 260.41666],\n dtype=float32)Latitude(Latitude)float3215.92 16.08 16.25 ... 35.58 35.75standard_name :latitudeunits :degrees_northpoint_spacing :evenlong_name :latitudeaxis :Ybounds :Lat_boundsarray([15.916667, 16.083334, 16.25 , 16.416666, 16.583334, 16.75 ,\n 16.916666, 17.083334, 17.25 , 17.416666, 17.583334, 17.75 ,\n 17.916666, 18.083334, 18.25 , 18.416666, 18.583334, 18.75 ,\n 18.916666, 19.083334, 19.25 , 19.416666, 19.583334, 19.75 ,\n 19.916666, 20.083334, 20.25 , 20.416666, 20.583334, 20.75 ,\n 20.916666, 21.083334, 21.25 , 21.416666, 21.583334, 21.75 ,\n 21.916666, 22.083334, 22.25 , 22.416666, 22.583334, 22.75 ,\n 22.916666, 23.083334, 23.25 , 23.416666, 23.583334, 23.75 ,\n 23.916666, 24.083334, 24.25 , 24.416666, 24.583334, 24.75 ,\n 24.916666, 25.083334, 25.25 , 25.416666, 25.583334, 25.75 ,\n 25.916666, 26.083334, 26.25 , 26.416666, 26.583334, 26.75 ,\n 26.916666, 27.083334, 27.25 , 27.416666, 27.583334, 27.75 ,\n 27.916666, 28.083334, 28.25 , 28.416666, 28.583334, 28.75 ,\n 28.916666, 29.083334, 29.25 , 29.416666, 29.583334, 29.75 ,\n 29.916666, 30.083334, 30.25 , 30.416666, 30.583334, 30.75 ,\n 30.916666, 31.083334, 31.25 , 31.416666, 31.583334, 31.75 ,\n 31.916666, 32.083332, 32.25 , 32.416668, 32.583332, 32.75 ,\n 32.916668, 33.083332, 33.25 , 33.416668, 33.583332, 33.75 ,\n 33.916668, 34.083332, 34.25 , 34.416668, 34.583332, 34.75 ,\n 34.916668, 35.083332, 35.25 , 35.416668, 35.583332, 35.75 ],\n dtype=float32)Time(Time)datetime64[ns]2021-07-05T12:00:00 ... 2021-09-...standard_name :timelong_name :Timebounds :Time_boundsaxis :Tarray(['2021-07-05T12:00:00.000000000', '2021-07-10T12:00:00.000000000',\n '2021-07-15T12:00:00.000000000', '2021-07-20T12:00:00.000000000',\n '2021-07-25T12:00:00.000000000', '2021-07-30T12:00:00.000000000',\n '2021-08-04T12:00:00.000000000', '2021-08-09T12:00:00.000000000',\n '2021-08-14T12:00:00.000000000', '2021-08-19T12:00:00.000000000',\n '2021-08-24T12:00:00.000000000', '2021-08-29T12:00:00.000000000',\n '2021-09-03T12:00:00.000000000', '2021-09-08T12:00:00.000000000',\n '2021-09-13T12:00:00.000000000', '2021-09-18T12:00:00.000000000',\n '2021-09-23T12:00:00.000000000', '2021-09-28T12:00:00.000000000'],\n dtype='datetime64[ns]')Indexes: (3)LongitudePandasIndexPandasIndex(Index([234.5833282470703, 234.75, 234.9166717529297,\n 235.0833282470703, 235.25, 235.4166717529297,\n 235.5833282470703, 235.75, 235.9166717529297,\n 236.0833282470703,\n ...\n 258.9166564941406, 259.0833435058594, 259.25,\n 259.4166564941406, 259.5833435058594, 259.75,\n 259.9166564941406, 260.0833435058594, 260.25,\n 260.4166564941406],\n dtype='float32', name='Longitude', length=156))LatitudePandasIndexPandasIndex(Index([15.916666984558105, 16.08333396911621, 16.25,\n 16.41666603088379, 16.58333396911621, 16.75,\n 16.91666603088379, 17.08333396911621, 17.25,\n 17.41666603088379,\n ...\n 34.25, 34.41666793823242, 34.58333206176758,\n 34.75, 34.91666793823242, 35.08333206176758,\n 35.25, 35.41666793823242, 35.58333206176758,\n 35.75],\n dtype='float32', name='Latitude', length=120))TimePandasIndexPandasIndex(DatetimeIndex(['2021-07-05 12:00:00', '2021-07-10 12:00:00',\n '2021-07-15 12:00:00', '2021-07-20 12:00:00',\n '2021-07-25 12:00:00', '2021-07-30 12:00:00',\n '2021-08-04 12:00:00', '2021-08-09 12:00:00',\n '2021-08-14 12:00:00', '2021-08-19 12:00:00',\n '2021-08-24 12:00:00', '2021-08-29 12:00:00',\n '2021-09-03 12:00:00', '2021-09-08 12:00:00',\n '2021-09-13 12:00:00', '2021-09-18 12:00:00',\n '2021-09-23 12:00:00', '2021-09-28 12:00:00'],\n dtype='datetime64[ns]', name='Time', freq=None))Attributes: (4)units :mlong_name :Sea Level Anomaly Estimatestandard_name :sea_surface_height_above_sea_levelalias :sea_surface_height_above_sea_level" }, { - "objectID": "tutorials/Sea_Level_Rise.html#recreate-the-sea-level-rise-infographic", - "href": "tutorials/Sea_Level_Rise.html#recreate-the-sea-level-rise-infographic", - "title": "Analyzing Sea Level Rise with NASA Earthdata", - "section": "Recreate the Sea Level Rise Infographic", - "text": "Recreate the Sea Level Rise Infographic\nFirst, we define a function that will calculate the area in 1/6 by 1/6 degree boxes in order to calculate the global mean of the SSH later.\n\ndef ssl_area(lats):\n \"\"\"\n Calculate the area associated with a 1/6 by 1/6 degree box at latitude specified in 'lats'.\n \n Parameter\n ==========\n lats: a list or numpy array of size N the latitudes of interest. \n \n Return\n =======\n out: Array (N) area values (unit: m^2)\n \"\"\"\n # Define WGS84 as CRS:\n geod = Geod(ellps='WGS84')\n dx=1/12.0\n # create a lambda function for calculating the perimeters of the boxes\n c_area=lambda lat: geod.polygon_area_perimeter(np.r_[-dx,dx,dx,-dx], lat+np.r_[-dx,-dx,dx,dx])[0]\n out=[]\n for lat in lats:\n out.append(c_area(lat))\n return np.array(out)\n\nLet’s use the function on our Sea Surface Height dataset.\n\n# note: they rotated the data in the last release, this operation used to be (1,-1)\nssh_area = ssl_area(ds_SSH.Latitude.data).reshape(-1,1)\nprint(ssh_area.shape)\n\n(960, 1)\n\n\nNext, we find and open the historic record dataset also using earthaccess and xarray.\n\nhistoric_ts_results = earthaccess.search_data(short_name='JPL_RECON_GMSL')\n\nGranules found: 1\n\n\n\nhistoric_ts=xr.open_mfdataset(earthaccess.open([historic_ts_results[0]]), engine='h5netcdf')\nhistoric_ts\n\n Opening 1 granules, approx size: 0.0 GB\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (time: 119)\nCoordinates:\n * time (time) datetime64[ns] ...\nData variables: (12/21)\n global_average_sea_level_change (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n global_average_sea_level_change_upper (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n global_average_sea_level_change_lower (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n glac_mean (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n glac_upper (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n glac_lower (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n ... ...\n global_average_thermosteric_sea_level_change (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n global_average_thermosteric_sea_level_change_upper (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n global_average_thermosteric_sea_level_change_lower (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n sum_of_contrib_processes_mean (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n sum_of_contrib_processes_upper (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\n sum_of_contrib_processes_lower (time) float32 dask.array<chunksize=(119,), meta=np.ndarray>\nAttributes: (12/42)\n title: Global sea-level changes and contributing proc...\n summary: This file contains reconstructed global-mean s...\n id: 10.5067/GMSLT-FJPL1\n naming_authority: gov.nasa.jpl\n source: Frederikse et al. The causes of sea-level rise...\n project: NASA sea-level change science team (N-SLCT)\n ... ...\n time_coverage_start: 1900-01-01\n time_coverage_end: 2018-12-31\n time_coverage_duration: P119Y\n time_coverage_resolution: P1Y\n date_created: 2020-07-28\n date_modified: 2020-09-14xarray.DatasetDimensions:time: 119Coordinates: (1)time(time)datetime64[ns]1900-06-15 ... 2018-06-15long_name :timestandard_name :timearray(['1900-06-15T00:00:00.000000000', '1901-06-15T00:00:00.000000000',\n '1902-06-15T00:00:00.000000000', '1903-06-15T00:00:00.000000000',\n '1904-06-15T00:00:00.000000000', '1905-06-15T00:00:00.000000000',\n '1906-06-15T00:00:00.000000000', '1907-06-15T00:00:00.000000000',\n '1908-06-15T00:00:00.000000000', '1909-06-15T00:00:00.000000000',\n '1910-06-15T00:00:00.000000000', '1911-06-15T00:00:00.000000000',\n '1912-06-15T00:00:00.000000000', '1913-06-15T00:00:00.000000000',\n '1914-06-15T00:00:00.000000000', '1915-06-15T00:00:00.000000000',\n '1916-06-15T00:00:00.000000000', '1917-06-15T00:00:00.000000000',\n '1918-06-15T00:00:00.000000000', '1919-06-15T00:00:00.000000000',\n '1920-06-15T00:00:00.000000000', '1921-06-15T00:00:00.000000000',\n '1922-06-15T00:00:00.000000000', '1923-06-15T00:00:00.000000000',\n '1924-06-15T00:00:00.000000000', '1925-06-15T00:00:00.000000000',\n '1926-06-15T00:00:00.000000000', '1927-06-15T00:00:00.000000000',\n '1928-06-15T00:00:00.000000000', '1929-06-15T00:00:00.000000000',\n '1930-06-15T00:00:00.000000000', '1931-06-15T00:00:00.000000000',\n '1932-06-15T00:00:00.000000000', '1933-06-15T00:00:00.000000000',\n '1934-06-15T00:00:00.000000000', '1935-06-15T00:00:00.000000000',\n '1936-06-15T00:00:00.000000000', '1937-06-15T00:00:00.000000000',\n '1938-06-15T00:00:00.000000000', '1939-06-15T00:00:00.000000000',\n '1940-06-15T00:00:00.000000000', '1941-06-15T00:00:00.000000000',\n '1942-06-15T00:00:00.000000000', '1943-06-15T00:00:00.000000000',\n '1944-06-15T00:00:00.000000000', '1945-06-15T00:00:00.000000000',\n '1946-06-15T00:00:00.000000000', '1947-06-15T00:00:00.000000000',\n '1948-06-15T00:00:00.000000000', '1949-06-15T00:00:00.000000000',\n '1950-06-15T00:00:00.000000000', '1951-06-15T00:00:00.000000000',\n '1952-06-15T00:00:00.000000000', '1953-06-15T00:00:00.000000000',\n '1954-06-15T00:00:00.000000000', '1955-06-15T00:00:00.000000000',\n '1956-06-15T00:00:00.000000000', '1957-06-15T00:00:00.000000000',\n '1958-06-15T00:00:00.000000000', '1959-06-15T00:00:00.000000000',\n '1960-06-15T00:00:00.000000000', '1961-06-15T00:00:00.000000000',\n '1962-06-15T00:00:00.000000000', '1963-06-15T00:00:00.000000000',\n '1964-06-15T00:00:00.000000000', '1965-06-15T00:00:00.000000000',\n '1966-06-15T00:00:00.000000000', '1967-06-15T00:00:00.000000000',\n '1968-06-15T00:00:00.000000000', '1969-06-15T00:00:00.000000000',\n '1970-06-15T00:00:00.000000000', '1971-06-15T00:00:00.000000000',\n '1972-06-15T00:00:00.000000000', '1973-06-15T00:00:00.000000000',\n '1974-06-15T00:00:00.000000000', '1975-06-15T00:00:00.000000000',\n '1976-06-15T00:00:00.000000000', '1977-06-15T00:00:00.000000000',\n '1978-06-15T00:00:00.000000000', '1979-06-15T00:00:00.000000000',\n '1980-06-15T00:00:00.000000000', '1981-06-15T00:00:00.000000000',\n '1982-06-15T00:00:00.000000000', '1983-06-15T00:00:00.000000000',\n '1984-06-15T00:00:00.000000000', '1985-06-15T00:00:00.000000000',\n '1986-06-15T00:00:00.000000000', '1987-06-15T00:00:00.000000000',\n '1988-06-15T00:00:00.000000000', '1989-06-15T00:00:00.000000000',\n '1990-06-15T00:00:00.000000000', '1991-06-15T00:00:00.000000000',\n '1992-06-15T00:00:00.000000000', '1993-06-15T00:00:00.000000000',\n '1994-06-15T00:00:00.000000000', '1995-06-15T00:00:00.000000000',\n '1996-06-15T00:00:00.000000000', '1997-06-15T00:00:00.000000000',\n '1998-06-15T00:00:00.000000000', '1999-06-15T00:00:00.000000000',\n '2000-06-15T00:00:00.000000000', '2001-06-15T00:00:00.000000000',\n '2002-06-15T00:00:00.000000000', '2003-06-15T00:00:00.000000000',\n '2004-06-15T00:00:00.000000000', '2005-06-15T00:00:00.000000000',\n '2006-06-15T00:00:00.000000000', '2007-06-15T00:00:00.000000000',\n '2008-06-15T00:00:00.000000000', '2009-06-15T00:00:00.000000000',\n '2010-06-15T00:00:00.000000000', '2011-06-15T00:00:00.000000000',\n '2012-06-15T00:00:00.000000000', '2013-06-15T00:00:00.000000000',\n '2014-06-15T00:00:00.000000000', '2015-06-15T00:00:00.000000000',\n '2016-06-15T00:00:00.000000000', '2017-06-15T00:00:00.000000000',\n '2018-06-15T00:00:00.000000000'], dtype='datetime64[ns]')Data variables: (21)global_average_sea_level_change(time)float32dask.array<chunksize=(119,), meta=np.ndarray>units :mmlong_name :Observed global-average sea level (mean value)valid_min :-1000000.0valid_max :1000000.0comment :Value is relative to 2002-2019 baselinestandard_name :global_average_sea_level_changecoverage_content_type :physicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\nglobal_average_sea_level_change_upper\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nObserved global-average sea level (upper bound)\n\nstandard_name :\n\nglobal_average_sea_level_change\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nglobal_average_sea_level_change_lower\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nObserved global-mean sea level (lower bound)\n\nstandard_name :\n\nglobal_average_sea_level_change\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nglac_mean\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGlacier contribution (mean value)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. The glacier term does not include peripheral glaciers of both ice sheets\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nglac_upper\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGlacier contribution (upper bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. The glacier term does not include peripheral glaciers of both ice sheets\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nglac_lower\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGlacier contribution (lower bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. The glacier term does not include peripheral glaciers of both ice sheets\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nGrIS_mean\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGreenland Ice Sheet contribution (mean value)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. This term includes glaciers in the Greenland periphery\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nGrIS_upper\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGreenland Ice Sheet contribution (upper bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. This term includes glaciers in the Greenland periphery\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nGrIS_lower\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGreenland Ice Sheet contribution (lower bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. This term includes glaciers in the Greenland periphery\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nAIS_mean\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nAntarctic Ice Sheet contribution (mean value)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline.\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nAIS_upper\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nAntarctic Ice Sheet contribution (upper bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline.\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nAIS_lower\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nAntarctic Ice Sheet contribution (lower bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline.\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\ntws_mean\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nTerrestrial water storage contribution (mean value)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. This term includes the effects for groundwater depletion, reservoir impoundment and non-anthropogenic tws changes\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\ntws_upper\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nTerrestrial water storage contribution (upper bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. This term includes the effects for groundwater depletion, reservoir impoundment and non-anthropogenic tws changes\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\ntws_lower\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nTerrestrial water storage contribution (lower bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline. This term includes the effects for groundwater depletion, reservoir impoundment and non-anthropogenic tws changes\n\nstandard_name :\n\nglobal_average_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nglobal_average_thermosteric_sea_level_change\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGlobal thermosteric contribution (mean value)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline\n\nstandard_name :\n\nglobal_average_thermosteric_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nglobal_average_thermosteric_sea_level_change_upper\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGlobal thermosteric contribution (upper bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline\n\nstandard_name :\n\nglobal_average_thermosteric_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nglobal_average_thermosteric_sea_level_change_lower\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nGlobal thermosteric contribution (lower bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline\n\nstandard_name :\n\nglobal_average_thermosteric_sea_level_change\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nsum_of_contrib_processes_mean\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nSum of contributing processes (mean value)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nsum_of_contrib_processes_upper\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nSum of contributing processes (upper bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nsum_of_contrib_processes_lower\n\n\n(time)\n\n\nfloat32\n\n\ndask.array<chunksize=(119,), meta=np.ndarray>\n\n\n\n\nunits :\n\nmm\n\nlong_name :\n\nSum of contributing processes (lower bound)\n\nvalid_min :\n\n-1000000.0\n\nvalid_max :\n\n1000000.0\n\ncomment :\n\nValue is relative to 2002-2019 baseline\n\ncoverage_content_type :\n\nphysicalMeasurement\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n476 B\n476 B\n\n\nShape\n(119,)\n(119,)\n\n\nDask graph\n1 chunks in 2 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nIndexes: (1)timePandasIndexPandasIndex(DatetimeIndex(['1900-06-15', '1901-06-15', '1902-06-15', '1903-06-15',\n '1904-06-15', '1905-06-15', '1906-06-15', '1907-06-15',\n '1908-06-15', '1909-06-15',\n ...\n '2009-06-15', '2010-06-15', '2011-06-15', '2012-06-15',\n '2013-06-15', '2014-06-15', '2015-06-15', '2016-06-15',\n '2017-06-15', '2018-06-15'],\n dtype='datetime64[ns]', name='time', length=119, freq=None))Attributes: (42)title :Global sea-level changes and contributing processes over 1900-2018summary :This file contains reconstructed global-mean sea level evolution and the estimated contributing processes over 1900-2018. Reconstructed sea level is based on annual-mean tide-gauge observations and uses the virtual-station method to aggregrate the individual observations into a global estimate. The contributing processes consist of thermosteric changes, glacier mass changes, mass changes of the Greenland and Antarctic Ice Sheet, and terrestrial water storage changes. The glacier, ice sheet, and terrestrial water storage are estimated by combining GRACE observations (2003-2018) with long-term estimates from in-situ observations and models. Steric estimates are based on in-situ temperature profiles. The upper- and lower bound represent the 5 and 95 percent confidence level. The numbers are equal to the ones presented in Frederikse et al. The causes of sea-level rise since 1900, Nature, 2020, reformatted to meet the specifications of the JPL PO.DAACid :10.5067/GMSLT-FJPL1naming_authority :gov.nasa.jplsource :Frederikse et al. The causes of sea-level rise since 1900, Nature, 2020 https://doi.org/10.1038/s41586-020-2591-3project :NASA sea-level change science team (N-SLCT)program :NASA sea-level change science team (N-SLCT)institution :NASA Jet Propulsion Laboratory (JPL)references :https://doi.org/10.5067/GMSLT-FJPL1,https://doi.org/10.1038/s41586-020-2591-3acknowledgement :This research was carried out by the Jet Propulsion Laboratory, managed by the California Institute of Technology under a contract with the National Aeronautics and Space Administration.processing_level :4product_version :1.0license :Public Domainhistory :This version provides the data as presented in Frederikse et al. 2020.Conventions :CF-1.7,ACDD-1.3keywords :EARTH SCIENCE > CLIMATE INDICATORS > ATMOSPHERIC/OCEAN INDICATORS > SEA LEVEL RISEkeywords_vocabulary :GCMD Science Keywordsstandard_name_vocabulary :CF Standard Name Table v27creator_name :Thomas Frederiksecreator_type :personcreator_url :https://sealevel.nasa.govcreator_email :thomas.frederikse@jpl.nasa.govcreator_institution :NASA Jet Propulsion Laboratory (JPL)contributor_name :Thomas Frederikse, Felix Landerer, Lambert Caron, Surendra Adhikari, David Parkes, Vincent Humphrey, Sönke Dangendorf, Peter Hogarth, Laure Zanna, Lijing Cheng, Yun-Hao Wucontributor_role :main author,author,author,author,author,author,author,author,author,author,authorpublisher_name :Physical Oceanography Distributed Active Archive Center (PO.DAAC)publisher_type :grouppublisher_url :https://podaac.jpl.nasa.govpublisher_email :podaac@podaac.jpl.nasa.govpublisher_institution :NASA Jet Propulsion Laboratory (JPL)geospatial_lat_min :-90.0geospatial_lat_max :90.0geospatial_lat_units :degrees_northgeospatial_lon_min :-180.0geospatial_lon_max :180.0geospatial_lon_units :degrees_easttime_coverage_start :1900-01-01time_coverage_end :2018-12-31time_coverage_duration :P119Ytime_coverage_resolution :P1Ydate_created :2020-07-28date_modified :2020-09-14\n\n\n\nLet’s Plot!\n\n%%time\n\nplt.rcParams[\"figure.figsize\"] = (16,4)\n\nfig, axs = plt.subplots()\nplt.grid(True)\n\n#function to get the global mean for plotting\ndef global_mean(SLA, **kwargs):\n dout=((SLA*ssh_area).sum()/(SLA/SLA*ssh_area).sum())*1000\n return dout\n\nresult = ds_SSH.SLA.groupby('Time').apply(global_mean)\n\nplt.xlabel('Time (year)',fontsize=16)\nplt.ylabel('Global Mean SLA (meter)',fontsize=12)\nplt.grid(True)\n\nresult.plot(ax=axs, color=\"orange\", marker=\"o\", label='Satellite Record')\n\nhistoric_ts['global_average_sea_level_change'].plot(ax=axs, label='Historical in-situ record', color=\"lightblue\")\n\nplt.legend()\nplt.show()\n\n\n\n\nCPU times: user 3.33 s, sys: 1.04 s, total: 4.37 s\nWall time: 4.05 s\n\n\n\nThis Data Story used content from Luis Lopez’s earthaccess tutorial, which was based on Jinbo Wang’s Earthdata Webinar tutorial." + "objectID": "how-tos/using-s3-storage.html#home-directory", + "href": "how-tos/using-s3-storage.html#home-directory", + "title": "Using S3 Bucket Storage in NASA-Openscapes Hub", + "section": "", + "text": "Imagining this ds_subset object is now an important intermediate dataset, or the result of a complex analysis and we want to save it. Our default action might be to just save it to our HOME directory. This is simple, but we want to avoid this as it incurs significant storage costs, and using this data later will be slow.\nds_subset.to_netcdf(\"test.nc\") # avoid writing to home directory like this" }, { - "objectID": "tutorials/matlab.html", - "href": "tutorials/matlab.html", - "title": "MATLAB Access Single NASA EarthData L2 NetCDF", + "objectID": "how-tos/using-s3-storage.html#use-the-s3fs-package-to-interact-with-our-s3-bucket.", + "href": "how-tos/using-s3-storage.html#use-the-s3fs-package-to-interact-with-our-s3-bucket.", + "title": "Using S3 Bucket Storage in NASA-Openscapes Hub", "section": "", - "text": "This tutorial was written by Erin Robinson, Luis Lopez (NASA Openscapes) and Cedric Wannaz, Lisa Kempler (Mathworks). Also available as a 9-minute Video walk-through." + "text": "s3fs is a Python library that allows us to interact with S3 objects in a file-system like manner.\n\n# Create a S3FileSystem class\ns3 = s3fs.S3FileSystem()\n\n# Get scratch and persistent buckets\nscratch = os.environ[\"SCRATCH_BUCKET\"]\npersistent = os.environ[\"PERSISTENT_BUCKET\"]\n\nprint(scratch)\nprint(persistent)\n\ns3://openscapeshub-scratch/ateucher\ns3://openscapeshub-persistent/ateucher\n\n\nOur user-specific directories in the two buckets aren’t actually created until we put something in them, so if we try to check for their existence or list their contents before they are created, we will get an error. We will use the S3FileSystem.touch() method to place a simple empty file called .placeholder in each one to bring them into existence.\n\ns3.touch(f\"{scratch}/.placeholder\")\n\ns3.ls(scratch)\n\n['openscapeshub-scratch/ateucher/.placeholder']\n\n\nand in our persistent bucket:\n\ns3.touch(f\"{persistent}/.placeholder\")\n\ns3.ls(persistent)\n\n['openscapeshub-persistent/ateucher/.placeholder']\n\n\n(Note that adding these placeholders isn’t strictly necessary, as the first time you write anything to these buckets they will be created.)" }, { - "objectID": "tutorials/matlab.html#launch-matlab-in-2i2c-jupyterhub", - "href": "tutorials/matlab.html#launch-matlab-in-2i2c-jupyterhub", - "title": "MATLAB Access Single NASA EarthData L2 NetCDF", - "section": "Launch MATLAB in 2i2c JupyterHub", - "text": "Launch MATLAB in 2i2c JupyterHub\nYou’re able to use MATLAB in the 2i2c JupyterHub; you are able to bring your own license. If you’d like to access MATLAB from the Amazon Web Services Console, see details at the bottom of this page.\n\n\n\nSelect MATLAB from the Server Options and press Start\n\n\n\n\n\nLaunch MATLAB from the JupyterHub interface by double-clicking\n\n\n\n\n\nEnter your MATLAB login information. NOTE: You will also need to be able to access your MATLAB account via MATLAB Online." + "objectID": "how-tos/using-s3-storage.html#save-dataset-as-netcdf-file-in-scratch-bucket", + "href": "how-tos/using-s3-storage.html#save-dataset-as-netcdf-file-in-scratch-bucket", + "title": "Using S3 Bucket Storage in NASA-Openscapes Hub", + "section": "", + "text": "Next we can save ds_subset as a netcdf file in our scratch bucket. This involves writing to a temporary directory first, and then moving that to the SCRATCH bucket:\n\n# Where we want to store it:\nscratch_nc_file_path = f\"{scratch}/test123.nc\"\n\n# Create a temporary intermediate file and save it to the bucket\nwith tempfile.NamedTemporaryFile(suffix = \".nc\") as tmp:\n ds_subset.to_netcdf(tmp.name) # save it to a temporary file\n s3.put(tmp.name, scratch_nc_file_path) # move that file to the scratch bucket\n\n# Ensure the file is there\ns3.ls(scratch)\n\n['openscapeshub-scratch/ateucher/.placeholder',\n 'openscapeshub-scratch/ateucher/test123.nc']\n\n\nAnd we can open it to ensure it worked:\n\nds_from_scratch = xr.open_dataarray(s3.open(scratch_nc_file_path))\n\nds_from_scratch\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray 'SLA' (Time: 18, Latitude: 120, Longitude: 156)> Size: 1MB\n[336960 values with dtype=float32]\nCoordinates:\n * Longitude (Longitude) float32 624B 234.6 234.8 234.9 ... 260.1 260.2 260.4\n * Latitude (Latitude) float32 480B 15.92 16.08 16.25 ... 35.42 35.58 35.75\n * Time (Time) datetime64[ns] 144B 2021-07-05T12:00:00 ... 2021-09-28T...\nAttributes:\n units: m\n long_name: Sea Level Anomaly Estimate\n standard_name: sea_surface_height_above_sea_level\n alias: sea_surface_height_above_sea_levelxarray.DataArray'SLA'Time: 18Latitude: 120Longitude: 156...[336960 values with dtype=float32]Coordinates: (3)Longitude(Longitude)float32234.6 234.8 234.9 ... 260.2 260.4standard_name :longitudeunits :degrees_eastpoint_spacing :evenlong_name :longitudeaxis :Xbounds :Lon_boundsarray([234.58333, 234.75 , 234.91667, 235.08333, 235.25 , 235.41667,\n 235.58333, 235.75 , 235.91667, 236.08333, 236.25 , 236.41667,\n 236.58333, 236.75 , 236.91667, 237.08333, 237.25 , 237.41667,\n 237.58333, 237.75 , 237.91667, 238.08333, 238.25 , 238.41667,\n 238.58333, 238.75 , 238.91667, 239.08333, 239.25 , 239.41667,\n 239.58333, 239.75 , 239.91667, 240.08333, 240.25 , 240.41667,\n 240.58333, 240.75 , 240.91667, 241.08333, 241.25 , 241.41667,\n 241.58333, 241.75 , 241.91667, 242.08333, 242.25 , 242.41667,\n 242.58333, 242.75 , 242.91667, 243.08333, 243.25 , 243.41667,\n 243.58333, 243.75 , 243.91667, 244.08333, 244.25 , 244.41667,\n 244.58333, 244.75 , 244.91667, 245.08333, 245.25 , 245.41667,\n 245.58333, 245.75 , 245.91667, 246.08333, 246.25 , 246.41667,\n 246.58333, 246.75 , 246.91667, 247.08333, 247.25 , 247.41667,\n 247.58333, 247.75 , 247.91667, 248.08333, 248.25 , 248.41667,\n 248.58333, 248.75 , 248.91667, 249.08333, 249.25 , 249.41667,\n 249.58333, 249.75 , 249.91667, 250.08333, 250.25 , 250.41667,\n 250.58333, 250.75 , 250.91667, 251.08333, 251.25 , 251.41667,\n 251.58333, 251.75 , 251.91667, 252.08333, 252.25 , 252.41667,\n 252.58333, 252.75 , 252.91667, 253.08333, 253.25 , 253.41667,\n 253.58333, 253.75 , 253.91667, 254.08333, 254.25 , 254.41667,\n 254.58333, 254.75 , 254.91667, 255.08333, 255.25 , 255.41667,\n 255.58333, 255.75 , 255.91667, 256.08334, 256.25 , 256.41666,\n 256.58334, 256.75 , 256.91666, 257.08334, 257.25 , 257.41666,\n 257.58334, 257.75 , 257.91666, 258.08334, 258.25 , 258.41666,\n 258.58334, 258.75 , 258.91666, 259.08334, 259.25 , 259.41666,\n 259.58334, 259.75 , 259.91666, 260.08334, 260.25 , 260.41666],\n dtype=float32)Latitude(Latitude)float3215.92 16.08 16.25 ... 35.58 35.75standard_name :latitudeunits :degrees_northpoint_spacing :evenlong_name :latitudeaxis :Ybounds :Lat_boundsarray([15.916667, 16.083334, 16.25 , 16.416666, 16.583334, 16.75 ,\n 16.916666, 17.083334, 17.25 , 17.416666, 17.583334, 17.75 ,\n 17.916666, 18.083334, 18.25 , 18.416666, 18.583334, 18.75 ,\n 18.916666, 19.083334, 19.25 , 19.416666, 19.583334, 19.75 ,\n 19.916666, 20.083334, 20.25 , 20.416666, 20.583334, 20.75 ,\n 20.916666, 21.083334, 21.25 , 21.416666, 21.583334, 21.75 ,\n 21.916666, 22.083334, 22.25 , 22.416666, 22.583334, 22.75 ,\n 22.916666, 23.083334, 23.25 , 23.416666, 23.583334, 23.75 ,\n 23.916666, 24.083334, 24.25 , 24.416666, 24.583334, 24.75 ,\n 24.916666, 25.083334, 25.25 , 25.416666, 25.583334, 25.75 ,\n 25.916666, 26.083334, 26.25 , 26.416666, 26.583334, 26.75 ,\n 26.916666, 27.083334, 27.25 , 27.416666, 27.583334, 27.75 ,\n 27.916666, 28.083334, 28.25 , 28.416666, 28.583334, 28.75 ,\n 28.916666, 29.083334, 29.25 , 29.416666, 29.583334, 29.75 ,\n 29.916666, 30.083334, 30.25 , 30.416666, 30.583334, 30.75 ,\n 30.916666, 31.083334, 31.25 , 31.416666, 31.583334, 31.75 ,\n 31.916666, 32.083332, 32.25 , 32.416668, 32.583332, 32.75 ,\n 32.916668, 33.083332, 33.25 , 33.416668, 33.583332, 33.75 ,\n 33.916668, 34.083332, 34.25 , 34.416668, 34.583332, 34.75 ,\n 34.916668, 35.083332, 35.25 , 35.416668, 35.583332, 35.75 ],\n dtype=float32)Time(Time)datetime64[ns]2021-07-05T12:00:00 ... 2021-09-...standard_name :timelong_name :Timebounds :Time_boundsaxis :Tarray(['2021-07-05T12:00:00.000000000', '2021-07-10T12:00:00.000000000',\n '2021-07-15T12:00:00.000000000', '2021-07-20T12:00:00.000000000',\n '2021-07-25T12:00:00.000000000', '2021-07-30T12:00:00.000000000',\n '2021-08-04T12:00:00.000000000', '2021-08-09T12:00:00.000000000',\n '2021-08-14T12:00:00.000000000', '2021-08-19T12:00:00.000000000',\n '2021-08-24T12:00:00.000000000', '2021-08-29T12:00:00.000000000',\n '2021-09-03T12:00:00.000000000', '2021-09-08T12:00:00.000000000',\n '2021-09-13T12:00:00.000000000', '2021-09-18T12:00:00.000000000',\n '2021-09-23T12:00:00.000000000', '2021-09-28T12:00:00.000000000'],\n dtype='datetime64[ns]')Indexes: (3)LongitudePandasIndexPandasIndex(Index([234.5833282470703, 234.75, 234.9166717529297,\n 235.0833282470703, 235.25, 235.4166717529297,\n 235.5833282470703, 235.75, 235.9166717529297,\n 236.0833282470703,\n ...\n 258.9166564941406, 259.0833435058594, 259.25,\n 259.4166564941406, 259.5833435058594, 259.75,\n 259.9166564941406, 260.0833435058594, 260.25,\n 260.4166564941406],\n dtype='float32', name='Longitude', length=156))LatitudePandasIndexPandasIndex(Index([15.916666984558105, 16.08333396911621, 16.25,\n 16.41666603088379, 16.58333396911621, 16.75,\n 16.91666603088379, 17.08333396911621, 17.25,\n 17.41666603088379,\n ...\n 34.25, 34.41666793823242, 34.58333206176758,\n 34.75, 34.91666793823242, 35.08333206176758,\n 35.25, 35.41666793823242, 35.58333206176758,\n 35.75],\n dtype='float32', name='Latitude', length=120))TimePandasIndexPandasIndex(DatetimeIndex(['2021-07-05 12:00:00', '2021-07-10 12:00:00',\n '2021-07-15 12:00:00', '2021-07-20 12:00:00',\n '2021-07-25 12:00:00', '2021-07-30 12:00:00',\n '2021-08-04 12:00:00', '2021-08-09 12:00:00',\n '2021-08-14 12:00:00', '2021-08-19 12:00:00',\n '2021-08-24 12:00:00', '2021-08-29 12:00:00',\n '2021-09-03 12:00:00', '2021-09-08 12:00:00',\n '2021-09-13 12:00:00', '2021-09-18 12:00:00',\n '2021-09-23 12:00:00', '2021-09-28 12:00:00'],\n dtype='datetime64[ns]', name='Time', freq=None))Attributes: (4)units :mlong_name :Sea Level Anomaly Estimatestandard_name :sea_surface_height_above_sea_levelalias :sea_surface_height_above_sea_level\n\n\n\nds_from_scratch.hvplot.image(x='Longitude', y='Latitude', cmap='RdBu', clim=(-0.5, 0.5), title=\"Sea Level Anomaly Estimate (m)\")" }, { - "objectID": "tutorials/matlab.html#credentials", - "href": "tutorials/matlab.html#credentials", - "title": "MATLAB Access Single NASA EarthData L2 NetCDF", - "section": "Credentials", - "text": "Credentials\nDirect S3 access is achieved by passing NASA supplied temporary credentials to AWS so we can interact with S3 objects from applicable EarthData Cloud buckets. For now, each NASA DAAC has different AWS credentials endpoints. Below are some of the credential endpoints to various DAACs:\n\nPODAAC: https://archive.podaac.earthdata.nasa.gov/s3credentials\nGESDISC: https://data.gesdisc.earthdata.nasa.gov/s3credentials\nLPDAAC: https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials\nORNLDAAC: https://data.ornldaac.earthdata.nasa.gov/s3credentials\nGHRC DAAC: https://data.ghrc.earthdata.nasa.gov/s3credentials\nNSIDC: https://data.nsidc.earthdatacloud.nasa.gov/s3credentials\n\nYou will need your EarthData login to access these links. Create an account here: https://urs.earthdata.nasa.gov/users/new." + "objectID": "how-tos/using-s3-storage.html#move-data-to-the-persistent-bucket", + "href": "how-tos/using-s3-storage.html#move-data-to-the-persistent-bucket", + "title": "Using S3 Bucket Storage in NASA-Openscapes Hub", + "section": "", + "text": "If we decide this is a file we want to keep around for a longer time period, we can move it to our persistent bucket. We can even make a subdirectory in our persistent bucket to keep us organized:\n\npersistent_dest_dir = f\"{persistent}/my-analysis-data/\"\n\n# Make directory in persistent bucket\ns3.mkdir(persistent_dest_dir)\n\n# Move the file\ns3.mv(scratch_nc_file_path, persistent_dest_dir)\n\n# Check the scratch and persistent bucket listings:\ns3.ls(scratch)\n\n['openscapeshub-scratch/ateucher/.placeholder']\n\n\n\ns3.ls(persistent)\n\n['openscapeshub-persistent/ateucher/.placeholder',\n 'openscapeshub-persistent/ateucher/my-analysis-data']\n\n\n\ns3.ls(persistent_dest_dir)\n\n['openscapeshub-persistent/ateucher/my-analysis-data/test123.nc']" }, { - "objectID": "tutorials/matlab.html#prerequisites", - "href": "tutorials/matlab.html#prerequisites", - "title": "MATLAB Access Single NASA EarthData L2 NetCDF", - "section": "Prerequisites", - "text": "Prerequisites\nIf you’d rather watch these instructions, there is a short demo video here: https://drive.google.com/drive/folders/1en_SHtjtF4pUt4WU0SzyV16SEegjiwrT?usp=sharing\n\nSave the following function in a text file called loadAWSCredentials.m to your MATLAB working folder or to any folder that is on MATLAB search path.\n\nfunction s3Credentials = loadAWSCredentials(daacCredentialsEndpoint, login, password)\n if nargin < 2 || isempty(login)\n login = getenv('EARTHDATA_LOGIN') ;\n end\n if nargin < 3 || isempty(password)\n password = getenv('EARTHDATA_PASSWORD') ;\n end\n\n % Get S3 credentials from EarthData\n opts = weboptions('ContentType', 'json', 'HeaderFields', ...\n {'Authorization', ['Basic ',matlab.net.base64encode([login,':',password])]});\n s3Credentials = webread(daacCredentialsEndpoint, opts) ;\n\n % Set relevant environment variables with AWS credentials/region\n setenv('AWS_ACCESS_KEY_ID', s3Credentials.accessKeyId) ;\n setenv('AWS_SECRET_ACCESS_KEY', s3Credentials.secretAccessKey) ;\n setenv('AWS_SESSION_TOKEN', s3Credentials.sessionToken) ;\n setenv('AWS_DEFAULT_REGION', 'us-west-2') ;\nend\n\nDefine environment variables that specify your EarthData credentials by executing the following (updated with your credentials):\n\nsetenv(\"EARTHDATA_LOGIN\",\"your_earthdata_login\");\nsetenv(\"EARTHDATA_PASSWORD\",\"your_earthdata_password\");\nThis can be done manually in the command window, in your MATLAB startup script, or in any MATLAB function/script that is executed before calling loadAWSCredentials. We recommend not to do it from your project main script, so you can share it without divulging your credentials.\n\nSearch NASA EarthData and find the S3 link you want to access." + "objectID": "how-tos/using-s3-storage.html#move-existing-data-from-home-to-persistent_bucket", + "href": "how-tos/using-s3-storage.html#move-existing-data-from-home-to-persistent_bucket", + "title": "Using S3 Bucket Storage in NASA-Openscapes Hub", + "section": "", + "text": "You may already have some data in your HOME directory that you would like to move out to a persistent bucket. You can do that using the awsv2 s3 command line tool, which is already installed on the hub. You can open a terminal from the Hub Launcher - it will open in your HOME directory. You can then use the awsv2 s3 mv command to move a file to your bucket.\n\n\n$ awsv2 s3 mv my-big-file.nc $PERSISTENT_BUCKET/ # The trailing slash is important here\nmove: ./my-big-file.nc to s3://openscapeshub-persistent/ateucher/my-big-file.nc\n\n\n\nList the contents of the local results-data directory:\n$ ls results-data/\nmy-big-file1.nc my-big-file2.nc\nUse awsv2 s3 mv with the --recursive flag to move all files in a directory to a new directory in PERSISTENT_BUCKET\n$ awsv2 s3 mv --recursive results-data $PERSISTENT_BUCKET/results-data/\nmove: results-data/my-big-file1.nc to s3://openscapeshub-persistent/ateucher/results-data/my-big-file1.nc\nmove: results-data/my-big-file2.nc to s3://openscapeshub-persistent/ateucher/results-data/my-big-file2.nc" }, { - "objectID": "tutorials/matlab.html#accessing-data", - "href": "tutorials/matlab.html#accessing-data", - "title": "MATLAB Access Single NASA EarthData L2 NetCDF", - "section": "Accessing data", - "text": "Accessing data\nMATLAB HDF5 interfaces can access netCDF4 files stored in S3/Azure, while the netCDF interfaces cannot. The example below shows how to access a netCDF file using the HDF5 interfaces. SEe also this video: MATLAB and HDF5: Compression, Cloud, and Community – Ellen Johnson, MathWorks.\nMATLAB provides two interfaces for accessing HDF5 datasets:\n\nHigh-level access functions make it easy to read and view data in an HDF5 file or write a variable from the MATLAB® workspace into an HDF5 file\nLow-level functions in the HDF5 library packages provide direct access to the more than 300 functions in the HDF5 C library from the HDF Group.\n\nLow-level functions allow you to access advanced HDF5 functionality and bypasses I/O overhead in the high-level interface, but has a substantially higher learning curve unless already familiar with the HDF5 C API.\n\nExample\n\nFind the relevant DAAC/endpoint for the dataset that you want to access. For the following dataset:\n\ns3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20100619062008-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\nThe relevant DAAC is PODAAC and the relevant endpoint is:\nhttps://archive.podaac.earthdata.nasa.gov/s3credentials\n\nSet AWS credentials using the loadAWSCredentials function, passing the relevant endpoint:\n\ndaacCredentialsEndpoint = \"https://archive.podaac.earthdata.nasa.gov/s3credentials\";\nloadAWSCredentials(daacCredentialsEndpoint);\nThese credentials are temporary; they expire every 30 minutes or so and must be refreshed periodically.\n\nAccess data Define path of relevant file and dataset:\n\nFILE_NAME = \"s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20100619062008-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\";\nDATASET_NAME = \"sea_surface_temperature_4um\";\nDATASET_PATH = \"/\" + DATASET_NAME ;\nFirst, check that you can access the dataset:\nh5disp(FILE_NAME);\nIf credentials and paths are correct, h5disp will display the structure of the data file.\n\n\nUsing MATLAB high-level HDF5 interface\nunits = h5readatt(FILE_NAME,DATASET_PATH,\"units\");\nfill_value = h5readatt(FILE_NAME,DATASET_PATH,\"_FillValue\");\nname = h5readatt(FILE_NAME,DATASET_PATH,\"long_name\");\ndata = h5read(FILE_NAME,DATASET_PATH);\ndata(data == fill_value) = NaN;\nlat = h5read(FILE_NAME,\"/lat\");\nlon = h5read(FILE_NAME,\"/lon\");\nBuild a map using contour:\ncontour(lon,lat,data);\ntitle(sprintf(\"%s [%s]\",name,units));\n\n\n\nMATLAB example map\n\n\n\n\nUsing MATLAB low-level HDF5 interface\n% Open file and root group\nfile_id = H5F.open(FILE_NAME,\"H5F_ACC_RDONLY\",\"H5P_DEFAULT\");\nroot_id = H5G.open(file_id,\"/\") ;\n\n% Open/read sea surface temperature (SST) dataset\ndata_id = H5D.open(file_id,DATASET_PATH);\ndata=H5D.read(data_id,\"H5T_NATIVE_DOUBLE\",\"H5S_ALL\",\"H5S_ALL\",\"H5P_DEFAULT\");\n\n% Open/read/close SST units attribute\nattr_id = H5A.open_by_name(root_id,DATASET_NAME,\"units\");\nunits = H5A.read(attr_id,\"H5ML_DEFAULT\");\nH5A.close(attr_id);\n\n% Open/read/close SST long_name attribute\nattr_id = H5A.open_by_name(root_id,DATASET_NAME,\"long_name\");\nlong_name = H5A.read(attr_id,\"H5ML_DEFAULT\");\nH5A.close(attr_id);\n\n% Open/read/close SST _FillValue attribute\nattr_id = H5A.open_by_name(root_id,DATASET_NAME,\"_FillValue\");\nfillvalue = H5A.read(attr_id,\"H5T_NATIVE_DOUBLE\");\nH5A.close(attr_id);\n\n% Close SST dataset and root group\nH5D.close(data_id);\nH5G.close(root_id);\n\n% Open/read/close latitude dataset\nlat_id = H5D.open(file_id,\"lat\");\nlat = H5D.read(lat_id,\"H5T_NATIVE_DOUBLE\",\"H5S_ALL\",\"H5S_ALL\",\"H5P_DEFAULT\");\nH5D.close(lat_id);\n\n% Open/read/close longitude dataset\nlon_id = H5D.open(file_id,\"lon\");\nlon = H5D.read(lon_id,\"H5T_NATIVE_DOUBLE\",\"H5S_ALL\",\"H5S_ALL\",\"H5P_DEFAULT\");\nH5D.close(lon_id);\n\n% Close file\nH5F.close(file_id);\n\n% Replace fill value with NaN\ndata(data==fillvalue) = NaN;\n\n% Plot contour map\ncontour(lon,lat,data);\ntitle(sprintf(\"%s [%s]\",long_name{1},units{1}));" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#outline", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#outline", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Outline:", + "text": "Outline:\n\nIntroduction to NASA Earthdata’s move to the cloud\n\nBackground and motivation\nEnabling Open Science via “Analysis-in-Place”\nResources for cloud adopters: NASA Earthdata Openscapes\n\nNASA Earthdata discovery and access in the cloud\n\nPart 1: Explore Earthdata cloud data availablity\nPart 2: Working with Cloud-Optimized GeoTIFFs using NASA’s Common Metadata Repository Spatio-Temporal Assett Catalog (CMR-STAC)\nPart 3: Working with Zarr-formatted data using NASA’s Harmony cloud transformation service\n\n\n\nTutorial materials are adapted from repos on the NASA Openscapes public Github:\n\nThis notebook source code: update https://github.com/NASA-Openscapes/2021-Cloud-Workshop-AGU/tree/main/how-tos\nAlso available via online Quarto book: update https://nasa-openscapes.github.io/2021-Cloud-Workshop-AGU/" }, { - "objectID": "tutorials/matlab.html#launch-with-aws-console", - "href": "tutorials/matlab.html#launch-with-aws-console", - "title": "MATLAB Access Single NASA EarthData L2 NetCDF", - "section": "Launch with AWS Console", - "text": "Launch with AWS Console\nAn alternative is to launch with the AWS Console. This chapter is for research teams currently working in MATLAB with NASA EarthData and wanting to take advantage of doing analysis in the Cloud. Our initial focus is on Amazon Web Services (AWS). For general, background on the Cloud and NASA Earthdata’s migration to the Cloud, checkout earlier chapters of the cookbook [TO DO: Add a link and specific reference].\n\nPrerequisites\n\nMATLAB License - You must have access to a MATLAB license that allows for access to the cloud. To check your license, in the MATLAB command line enter:\nver -support\nAWS Account - For the current experiments, we will access the AWS console, so you need an AWS account. It is free to set up, but will require a credit card linked to the account. https://aws.amazon.com/\nAWS Region: We are working in AWS US-West-2 because NASA Earthdata is hosted in US-West-2. If there is an option to pick an AWS region, pick US-West-2.\nGitHub Account - If you want to use GitHub as part of your workflow and don’t already have an account, create a GitHub account: https://github.com/\nNASA Earthdata login - Create an account here: https://urs.earthdata.nasa.gov/users/new\nMicrosoft Remote Desktop - When we spin up the AWS virtual machine with MATLAB, we will need to have a remote desktop option. These how-to’s were done on a mac and we tested the free Microsoft Remote Desktop. Download for macs\n\n\n\nPrerequisite: AWS EC2 Key Pair\nThis isn’t included in the numbered prereq list above because it is a bit more involved than creating an account or knowing what region we work in. You need to create an SSH Key Pair in the region you want to work in. This is a one-time step that you won’t need to do each time you launch the stack.\n\nLog in to AWS\nIn the search box enter, ‘Create AWS Key Pair.’ Under features choose ‘Key Pairs’ (AWS help)\n\nCheck that the top bar says, ‘Oregon’ (AKA - US-West-2) and if not, click that down arrow to choose US-West-2. Then click the orange ‘Create key pair’ button.\n\nTroubleshooting tip: If you create the key outside of the region you want to work in, it will not show up when you launch the stack below.\nOn the form - give your key a name, choose the RSA and PEM options (these should be the defaults.)\n\nOk - now we are ready to start!\n\n\nCreating the AWS Stack with MATLAB\nNote: The first time I launched the AWS Stack it seemed to take for-ever. I thought “how could I possibly need to do this every time I want to use MATLAB in the cloud?” It does speed up, eventually it get’s a bit faster. I also have learned to plan a bit better - if I know I want to do some work, I get the launched, have coffee, and when I come back it’s ready to roll.\n\nFrom this MATLAB GitHub page click the release for 2022a under deployment steps.\nThis brings up a list of MATLAB on Amazon Web Services (Linux VM). Choose & click the ‘launch stack’ link for US-West-2.\n\nThis opens the ‘Quick create stack’ form based on the MATLAB template. That means that when you launch this stack it will come with MATLAB on the desktop. Fill out the form to create the AWS stack:\n\nGive the stack a name like ‘matlab-test’\nKeep the pre-filled options the same for now.\nRemote Access:\n\n“Allow Connections From:” You will need to know your IP address. You can google, “what’s my IP address?”\nEnter your IP address followed by a /32 like this -> [my.IP.address/32]\nIn the SSH Key Pair - the key pair you created above should show up in the drop down. If it doesn’t show up, see the troubleshooting tip.\nPick a remote password. This is not your AWS password or your GitHub password, this is the password that you will use to login with the microsoft remote desktop (username: ubuntu)\n\nNetwork configuration\n\nThere is one VPC option - choose that\nFor subnet - I pick the first one and it works. So pick the first option.\n\nAutoshutdown hasn’t worked for me so far, so for now I leave this set as never and delete the stack when I am finished.\nCheck the box that “I acknowledge that AWS CloudFormation might create IAM resources.”\nClick ‘Create stack’\nWait…. [~ 10 minutes]\n\nYou can check the status by clicking the refresh button on the right corner\n\n\n\n\nLaunch the AWS Stack with Microsoft Remote Desktop\n\nOnce the stack is created it will say ‘Create_complete’ on the left side.\n\nClick the outputs tab and copy the value text. It will start with ‘ec2-…’\nOpen Microsoft Remote Desktop\n\nClick the + to add a PC\nPaste the value text as the PC Name\nClick on the grey box of your new computer in the remote desktop window\nA login will pop up\n\n\nUsername is ubuntu\nPassword is the password you set in 3.3.3 above in the section on ‘Creating AWS Stack’\n\nA certificate message will pop up - say ok\nThe desktop will launch\n\nWait … [~2 mins]\n\n\n\nOpen MATLAB on Remote Desktop\n\nClick the MATLAB icon on the remote desktop\n\nWait … [~4 mins]\nLogin with your MATLAB credentials\nYou are in!\n\n\n\nShutdown your AWS Stack\nAfter each session you need to turn off the AWS Stack. If you forget this step and leave it running it is like keeping a computer on for the month. For the large instance it costs $0.5/day so it’s a few dollars a month.\n\nGo back to AWS\nSearch for stack\nClick on the name of your stack\nClick ‘Delete’\n\nConfirm the delete" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#the-nasa-earthdata-archive-continues-to-grow", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#the-nasa-earthdata-archive-continues-to-grow", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "The NASA Earthdata archive continues to grow", + "text": "The NASA Earthdata archive continues to grow\n\n\n\nEOSDIS Data Archive" }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html", - "title": "Observing Seasonality in Agricultural Areas", - "section": "", - "text": "This tutorial was developed to examine changes in enhanced vegetation index (EVI) over an agricultural region in northern California. The goal of the project is to observe HLS-derived mean EVI over these regions without downloading the entirety of the HLS source data. In this notebook we will extract an EVI timeseries from Harmonized Landsat Sentinel-2 (HLS) data in the Cloud using the earthaccess and rioxarray libraries. This tutorial will show how to find the HLS data available in the cloud for our specific time period, bands (layers), and region of interest. After finding the desired data, we will load subsets of the cloud optimized geotiffs (COGs) into a Jupyter Notebook directly from the cloud, and calculate EVI. After calculating EVI we will save and stack the time series, visualize it, and export a CSV of EVI statistics for the region.\nThe Harmonized Landsat Sentinel-2 (HLS) project produces seamless, harmonized surface reflectance data from the Operational Land Imager (OLI) and Multi-Spectral Instrument (MSI) aboard Landsat and Sentinel-2 Earth-observing satellites, respectively. The aim is to produce seamless products with normalized parameters, which include atmospheric correction, cloud and cloud-shadow masking, geographic co-registration and common gridding, normalized bidirectional reflectance distribution function, and spectral band adjustment. This will provide global observation of the Earth’s surface every 2-3 days with 30 meter spatial resolution. One of the major applications that will benefit from HLS is agriculture assessment and monitoring, which is used as the use case for this tutorial.\nNASA’s Land Processes Distributed Active Archive Center (LP DAAC) archives and distributes HLS products in the LP DAAC Cumulus cloud archive as Cloud Optimized GeoTIFFs (COG). This tutorial will demonstrate Because these data are stored as COGs, this tutorial will teach users how to load subsets of individual files into memory for just the bands you are interested in–a paradigm shift from the more common workflow where you would need to download a .zip/HDF file containing every band over the entire scene/tile. This tutorial covers how to process HLS data (calculate EVI), visualize, and “stack” the scenes over a region of interest into an xarray data array, calculate statistics for an EVI time series, and export as a comma-separated values (CSV) file–providing you with all of the information you need for your area of interest without having to download the source data file. The Enhanced Vegetation Index (EVI), is a vegetation index similar to NDVI that has been found to be more sensitive to ground cover below the vegetated canopy and saturates less over areas of dense green vegetation." + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#the-nasa-earthdata-cloud-evolution", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#the-nasa-earthdata-cloud-evolution", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "The NASA Earthdata Cloud Evolution", + "text": "The NASA Earthdata Cloud Evolution\n \n\nNASA Distributed Active Archive Centers (DAACs) are continuing to migrate data to the Earthdata Cloud\n\nSupporting increased data volume as new, high-resolution remote sensing missions launch in the coming years\nData hosted via Amazon Web Services, or AWS\nDAACs continuing to support tools, services, and tutorial resources for our user communities" }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#overview", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#overview", - "title": "Observing Seasonality in Agricultural Areas", - "section": "", - "text": "This tutorial was developed to examine changes in enhanced vegetation index (EVI) over an agricultural region in northern California. The goal of the project is to observe HLS-derived mean EVI over these regions without downloading the entirety of the HLS source data. In this notebook we will extract an EVI timeseries from Harmonized Landsat Sentinel-2 (HLS) data in the Cloud using the earthaccess and rioxarray libraries. This tutorial will show how to find the HLS data available in the cloud for our specific time period, bands (layers), and region of interest. After finding the desired data, we will load subsets of the cloud optimized geotiffs (COGs) into a Jupyter Notebook directly from the cloud, and calculate EVI. After calculating EVI we will save and stack the time series, visualize it, and export a CSV of EVI statistics for the region.\nThe Harmonized Landsat Sentinel-2 (HLS) project produces seamless, harmonized surface reflectance data from the Operational Land Imager (OLI) and Multi-Spectral Instrument (MSI) aboard Landsat and Sentinel-2 Earth-observing satellites, respectively. The aim is to produce seamless products with normalized parameters, which include atmospheric correction, cloud and cloud-shadow masking, geographic co-registration and common gridding, normalized bidirectional reflectance distribution function, and spectral band adjustment. This will provide global observation of the Earth’s surface every 2-3 days with 30 meter spatial resolution. One of the major applications that will benefit from HLS is agriculture assessment and monitoring, which is used as the use case for this tutorial.\nNASA’s Land Processes Distributed Active Archive Center (LP DAAC) archives and distributes HLS products in the LP DAAC Cumulus cloud archive as Cloud Optimized GeoTIFFs (COG). This tutorial will demonstrate Because these data are stored as COGs, this tutorial will teach users how to load subsets of individual files into memory for just the bands you are interested in–a paradigm shift from the more common workflow where you would need to download a .zip/HDF file containing every band over the entire scene/tile. This tutorial covers how to process HLS data (calculate EVI), visualize, and “stack” the scenes over a region of interest into an xarray data array, calculate statistics for an EVI time series, and export as a comma-separated values (CSV) file–providing you with all of the information you need for your area of interest without having to download the source data file. The Enhanced Vegetation Index (EVI), is a vegetation index similar to NDVI that has been found to be more sensitive to ground cover below the vegetated canopy and saturates less over areas of dense green vegetation." + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#nasa-earthdata-cloud-as-an-enabler-of-open-science", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#nasa-earthdata-cloud-as-an-enabler-of-open-science", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "NASA Earthdata Cloud as an enabler of Open Science", + "text": "NASA Earthdata Cloud as an enabler of Open Science\n\nReducing barriers to large-scale scientific research in the era of “big data”\nIncreasing community contributions with hands-on engagement\nPromoting reproducible and shareable workflows without relying on local storage systems\n\n\n\n\nOpen Data" }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#requirements", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#requirements", - "title": "Observing Seasonality in Agricultural Areas", - "section": "Requirements", - "text": "Requirements\n\nA NASA Earthdata Login account is required to download the data used in this tutorial. You can create an account at the link provided.\nYou will will also need to have a netrc file set up in your home directory in order to successfully run the code below. A code chunk in a later section provides a way to do this, or you can check out the setup_intstructions.md." + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#data-and-analysis-co-located-in-place", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#data-and-analysis-co-located-in-place", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Data and Analysis co-located “in place”", + "text": "Data and Analysis co-located “in place”\n\n\n\nEarthdata Cloud Paradigm" }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#learning-objectives", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#learning-objectives", - "title": "Observing Seasonality in Agricultural Areas", - "section": "Learning Objectives", - "text": "Learning Objectives\n\nHow to work with HLS Landsat (HLSL30.002) and Sentinel-2 (HLSS30.002) data products\n\nHow to query and subset HLS data using the earthaccess library\n\nHow to access and work with HLS data" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#building-nasa-earthdata-cloud-resources", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#building-nasa-earthdata-cloud-resources", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Building NASA Earthdata Cloud Resources", + "text": "Building NASA Earthdata Cloud Resources\nShow slide with 3 panels of user resources\nEmphasize that the following tutorials are short examples that were taken from the tutorial resources we have been building for our users" }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#data-used", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#data-used", - "title": "Observing Seasonality in Agricultural Areas", - "section": "Data Used", - "text": "Data Used\n\nDaily 30 meter (m) global HLS Sentinel-2 Multi-spectral Instrument Surface Reflectance - HLSS30.002\n\nThe HLSS30 product provides 30 m Nadir normalized Bidirectional Reflectance Distribution Function (BRDF)-Adjusted Reflectance (NBAR) and is derived from Sentinel-2A and Sentinel-2B MSI data products.\n\nScience Dataset (SDS) layers:\n\nB8A (NIR Narrow)\n\nB04 (Red)\n\nB02 (Blue)\n\nFmask (Quality)\n\n\n\nDaily 30 meter (m) global HLS Landsat-8 OLI Surface Reflectance - HLSL30.002\n\nThe HLSL30 product provides 30 m Nadir normalized Bidirectional Reflectance Distribution Function (BRDF)-Adjusted Reflectance (NBAR) and is derived from Landsat-8 OLI data products.\n\nScience Dataset (SDS) layers:\n\nB05 (NIR)\n\nB04 (Red)\n\nB02 (Blue)\n\nFmask (Quality)" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#nasa-earthdata-cloud-discovery-and-access-using-open-source-technologies", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#nasa-earthdata-cloud-discovery-and-access-using-open-source-technologies", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "NASA Earthdata Cloud: Discovery and access using open source technologies", + "text": "NASA Earthdata Cloud: Discovery and access using open source technologies\nThe following tutorial demonstrates several basic end-to-end workflows to interact with data “in-place” from the NASA Earthdata Cloud, accessing Amazon Web Services (AWS) Single Storage Solution (S3) data locations without the need to download data. While the data can be downloaded locally, the cloud offers the ability to scale compute resources to perform analyses over large areas and time spans, which is critical as data volumes continue to grow.\nAlthough the examples we’re working with in this notebook only focuses on a small time and area for demonstration purposes, this workflow can be modified and scaled up to suit a larger time range and region of interest.\n\nDatasets of interest:\n\nHarmonized Landsat Sentinel-2 (HLS) Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0 (L30) (10.5067/HLS/HLSL30.002)\n\nSurface reflectance (SR) and top of atmosphere (TOA) brightness data\nGlobal observations of the land every 2–3 days at 30-meter (m)\nCloud Optimized GeoTIFF (COG) format\n\nECCO Sea Surface Height - Daily Mean 0.5 Degree (Version 4 Release 4)(10.5067/ECG5D-SSH44).\n\nDaily-averaged dynamic sea surface height\nTime series of monthly NetCDFs on a 0.5-degree latitude/longitude grid." }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#tutorial-outline", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#tutorial-outline", - "title": "Observing Seasonality in Agricultural Areas", - "section": "Tutorial Outline", - "text": "Tutorial Outline\n\nGetting Started\n1.1 Import Packages\n1.2 EarthData Login\n\nFinding HLS Data\n\nAccessing HLS Cloud Optimized GeoTIFFs (COGs) from Earthdata Cloud 3.1 Subset by Band\n3.2 Load COGS into Memory\n3.3 Subset Spatially\n3.4 Apply Scale Factor\n\nProcessing HLS Data\n4.1 Calculate EVI\n4.2 Export to COG\n\nAutomation\n\nStacking HLS Data\n6.1 Open and Stack COGs\n6.2 Visualize Stacked Time Series\n\nExport Statistics" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#part-1-explore-data-hosted-in-the-earthdata-cloud", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#part-1-explore-data-hosted-in-the-earthdata-cloud", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Part 1: Explore Data hosted in the Earthdata Cloud", + "text": "Part 1: Explore Data hosted in the Earthdata Cloud\n\nEarthdata Search Demonstration\nFrom Earthdata Search https://search.earthdata.nasa.gov, use your Earthdata login credentials to log in. You can create an Earthdata Login account at https://urs.earthdata.nasa.gov.\nIn this example we are interested in the ECCO dataset, hosted by the PO.DAAC. This dataset is available from the NASA Earthdata Cloud archive hosted in AWS cloud.\nClick on the “Available from AWS Cloud” filter option on the left. Here, 39 matching collections were found with the ECCO monthly SSH search, and for the time period for year 2015. The latter can be done using the calendar icon on the left under the search box. Scroll down the list of returned matches until we see the dataset of interest, in this case ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4).\n\n\nView and Select Data Access Options\nClicking on the ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4) dataset, we now see a list of files (granules) that are part of the dataset (collection). We can click on the green + symbol to add a few files to our project. Here we added the first 3 listed for 2015. Then click on the green button towards the bottom that says “Download”. This will take us to another page with options to customize our download or access link(s).\n\n\n\nFigure caption: Select granules and click download" }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#getting-started", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#getting-started", - "title": "Observing Seasonality in Agricultural Areas", - "section": "1. Getting Started", - "text": "1. Getting Started\n\n1.1 Import Packages\nImport the required packages.\n\nimport os\nfrom datetime import datetime\nimport requests as r\nimport numpy as np\nimport pandas as pd\nimport geopandas as gp\nfrom skimage import io\nimport matplotlib.pyplot as plt\nfrom osgeo import gdal\nimport rasterio as rio\nimport xarray as xr\nimport rioxarray as rxr\nimport hvplot.xarray\nimport hvplot.pandas\nimport json\nimport panel as pn\nimport geoviews\nimport earthaccess\nfrom pprint import pprint\n\n\n\n\n\n\n\n\n\n\n\n\n\n1.2 Earthdata Login Authentication\nWe will use the earthaccess package for authentication. earthaccess can either createa a new local .netrc file to store credentials or validate that one exists already in you user profile. If you do not have a .netrc file, you will be prompted for your credentials and one will be created.\n\nauth = earthaccess.login()\n\nYou're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 11/06/2023\nUsing .netrc file for EDL" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#access-options", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#access-options", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Access Options", + "text": "Access Options\nSelect the “Direct Download” option to view Access options via Direct Download and from the AWS Cloud. Additional options to customize the data are also available for this dataset.\n\n\n\nFigure caption: Customize your download or access" }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#finding-hls-data-using-earthaccess", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#finding-hls-data-using-earthaccess", - "title": "Observing Seasonality in Agricultural Areas", - "section": "2. Finding HLS Data using earthaccess ", - "text": "2. Finding HLS Data using earthaccess \nTo find HLS data, we will use the earthaccess python library to search NASA’s Common Metadata Repository (CMR) for HLS data. We will use an geojson file containing our region of interest (ROI) to search for files that intersect. To do this, we will simplify it to a bounding box. Grab the bounding coordinates from the geopandas object after opening.\nFirst we will read in our geojson file using geopandas\n\nfield = gp.read_file('https://git.earthdata.nasa.gov/projects/LPDUR/repos/hls-tutorial/raw/Field_Boundary.geojson?at=refs%2Fheads%2Fmain')\n\nWe will use the total_bounds property to get the bounding box of our ROI, and add that to a python tuple, which is the expected data type for the bounding_box parameter earthaccess search_data.\n\nbbox = tuple(list(field.total_bounds))\nbbox\n\n(-122.0622682571411, 39.897234301806, -122.04918980598451, 39.91309383703065)\n\n\nWhen searching we can also search a specific time period of interest. Here we search from the beginning of May 2021 to the end of September 2021.\n\ntemporal = (\"2021-05-01T00:00:00\", \"2021-09-30T23:59:59\")\n\nSince the HLS collection contains to products, i.e. HLSL30 and HLSS30, we will include both short names. Search using our constraints and the count = 100 to limit our search to 100 results.\n\nresults = earthaccess.search_data(\n short_name=['HLSL30','HLSS30'],\n bounding_box=bbox,\n temporal=temporal, # 2021-07-15T00:00:00Z/2021-09-15T23:59:59Z\n count=100\n)\n\nGranules found: 78\n\n\nWe can preview these results in a pandas dataframe we want to check the metadata. Note we only show the first 5.\n\npd.json_normalize(results).head(5)\n\n\n\n\n\n\n\n\nsize\nmeta.concept-type\nmeta.concept-id\nmeta.revision-id\nmeta.native-id\nmeta.provider-id\nmeta.format\nmeta.revision-date\numm.TemporalExtent.RangeDateTime.BeginningDateTime\numm.TemporalExtent.RangeDateTime.EndingDateTime\n...\numm.CollectionReference.EntryTitle\numm.RelatedUrls\numm.DataGranule.DayNightFlag\numm.DataGranule.Identifiers\numm.DataGranule.ProductionDateTime\numm.DataGranule.ArchiveAndDistributionInformation\numm.Platforms\numm.MetadataSpecification.URL\numm.MetadataSpecification.Name\numm.MetadataSpecification.Version\n\n\n\n\n0\n198.882758\ngranule\nG2153255128-LPCLOUD\n1\nHLS.S30.T10TEK.2021122T185911.v2.0\nLPCLOUD\napplication/echo10+xml\n2021-10-29T10:47:01.310Z\n2021-05-02T19:13:19.930Z\n2021-05-02T19:13:19.930Z\n...\nHLS Sentinel-2 Multi-spectral Instrument Surfa...\n[{'URL': 'https://data.lpdaac.earthdatacloud.n...\nDay\n[{'Identifier': 'HLS.S30.T10TEK.2021122T185911...\n2021-10-29T10:44:28.000Z\n[{'Name': 'Not provided', 'SizeInBytes': 20854...\n[{'ShortName': 'Sentinel-2A', 'Instruments': [...\nhttps://cdn.earthdata.nasa.gov/umm/granule/v1.6.5\nUMM-G\n1.6.5\n\n\n1\n186.999763\ngranule\nG2153407526-LPCLOUD\n1\nHLS.S30.T10TEK.2021124T184919.v2.0\nLPCLOUD\napplication/echo10+xml\n2021-10-29T14:09:37.033Z\n2021-05-04T19:03:22.997Z\n2021-05-04T19:03:22.997Z\n...\nHLS Sentinel-2 Multi-spectral Instrument Surfa...\n[{'URL': 'https://data.lpdaac.earthdatacloud.n...\nDay\n[{'Identifier': 'HLS.S30.T10TEK.2021124T184919...\n2021-10-29T13:51:48.000Z\n[{'Name': 'Not provided', 'SizeInBytes': 19608...\n[{'ShortName': 'Sentinel-2B', 'Instruments': [...\nhttps://cdn.earthdata.nasa.gov/umm/granule/v1.6.5\nUMM-G\n1.6.5\n\n\n2\n199.237585\ngranule\nG2153881444-LPCLOUD\n1\nHLS.S30.T10TEK.2021127T185919.v2.0\nLPCLOUD\napplication/echo10+xml\n2021-10-30T00:59:05.213Z\n2021-05-07T19:13:19.594Z\n2021-05-07T19:13:19.594Z\n...\nHLS Sentinel-2 Multi-spectral Instrument Surfa...\n[{'URL': 'https://data.lpdaac.earthdatacloud.n...\nDay\n[{'Identifier': 'HLS.S30.T10TEK.2021127T185919...\n2021-10-29T21:47:54.000Z\n[{'Name': 'Not provided', 'SizeInBytes': 20891...\n[{'ShortName': 'Sentinel-2B', 'Instruments': [...\nhttps://cdn.earthdata.nasa.gov/umm/granule/v1.6.5\nUMM-G\n1.6.5\n\n\n3\n102.786560\ngranule\nG2144019307-LPCLOUD\n1\nHLS.L30.T10TEK.2021128T184447.v2.0\nLPCLOUD\napplication/echo10+xml\n2021-10-15T03:48:08.875Z\n2021-05-08T18:44:47.423Z\n2021-05-08T18:45:11.310Z\n...\nHLS Landsat Operational Land Imager Surface Re...\n[{'URL': 'https://data.lpdaac.earthdatacloud.n...\nDay\n[{'Identifier': 'HLS.L30.T10TEK.2021128T184447...\n2021-10-14T22:05:21.000Z\n[{'Name': 'Not provided', 'SizeInBytes': 10777...\n[{'ShortName': 'LANDSAT-8', 'Instruments': [{'...\nhttps://cdn.earthdata.nasa.gov/umm/granule/v1.6.5\nUMM-G\n1.6.5\n\n\n4\n186.955010\ngranule\nG2153591767-LPCLOUD\n1\nHLS.S30.T10TEK.2021129T184921.v2.0\nLPCLOUD\napplication/echo10+xml\n2021-10-29T16:27:32.621Z\n2021-05-09T19:03:25.151Z\n2021-05-09T19:03:25.151Z\n...\nHLS Sentinel-2 Multi-spectral Instrument Surfa...\n[{'URL': 'https://data.lpdaac.earthdatacloud.n...\nDay\n[{'Identifier': 'HLS.S30.T10TEK.2021129T184921...\n2021-10-29T16:13:58.000Z\n[{'Name': 'Not provided', 'SizeInBytes': 19603...\n[{'ShortName': 'Sentinel-2A', 'Instruments': [...\nhttps://cdn.earthdata.nasa.gov/umm/granule/v1.6.5\nUMM-G\n1.6.5\n\n\n\n\n5 rows × 24 columns\n\n\n\nWe can also preview each individual results by selecting it from the list. This will show the data links, and a browse image.\n\nresults[0]\n\n\n\n \n \n \n \n \n \n \n Data: HLS.S30.T10TEK.2021122T185911.v2.0.B10.tifHLS.S30.T10TEK.2021122T185911.v2.0.SZA.tifHLS.S30.T10TEK.2021122T185911.v2.0.B05.tifHLS.S30.T10TEK.2021122T185911.v2.0.VAA.tifHLS.S30.T10TEK.2021122T185911.v2.0.B04.tifHLS.S30.T10TEK.2021122T185911.v2.0.B01.tifHLS.S30.T10TEK.2021122T185911.v2.0.B07.tifHLS.S30.T10TEK.2021122T185911.v2.0.B02.tifHLS.S30.T10TEK.2021122T185911.v2.0.B08.tifHLS.S30.T10TEK.2021122T185911.v2.0.VZA.tifHLS.S30.T10TEK.2021122T185911.v2.0.B03.tifHLS.S30.T10TEK.2021122T185911.v2.0.B09.tifHLS.S30.T10TEK.2021122T185911.v2.0.B11.tifHLS.S30.T10TEK.2021122T185911.v2.0.B06.tifHLS.S30.T10TEK.2021122T185911.v2.0.B8A.tifHLS.S30.T10TEK.2021122T185911.v2.0.Fmask.tifHLS.S30.T10TEK.2021122T185911.v2.0.SAA.tifHLS.S30.T10TEK.2021122T185911.v2.0.B12.tif\n Size: 198.88 MB\n Cloud Hosted: True\n \n \n \n \n \n \n \n \n\n\nWe can grab all of the URLs for the data using list comprehension.\n\nhls_results_urls = [granule.data_links() for granule in results]\nprint(f\"There are {len(hls_results_urls)} granules and each granule contains {len(hls_results_urls[0])} assets. Below is an example of the assets printed for one granule:\")\npprint(hls_results_urls[0])\n\nThere are 78 granules and each granule contains 18 assets. Below is an example of the assets printed for one granule:\n['https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B10.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.SZA.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B05.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.VAA.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B01.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B07.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B02.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B08.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.VZA.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B03.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B09.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B11.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B06.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B8A.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.Fmask.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.SAA.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.B12.tif']\n\n\nWe can get the URLs for the browse images as well.\n\nbrowse_urls = [granule.dataviz_links()[0] for granule in results] # 0 retrieves only the https links\npprint(browse_urls[0:5]) # Print the first five results\n\n['https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSS30.020/HLS.S30.T10TEK.2021122T185911.v2.0/HLS.S30.T10TEK.2021122T185911.v2.0.jpg',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSS30.020/HLS.S30.T10TEK.2021124T184919.v2.0/HLS.S30.T10TEK.2021124T184919.v2.0.jpg',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSS30.020/HLS.S30.T10TEK.2021127T185919.v2.0/HLS.S30.T10TEK.2021127T185919.v2.0.jpg',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSL30.020/HLS.L30.T10TEK.2021128T184447.v2.0/HLS.L30.T10TEK.2021128T184447.v2.0.jpg',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSS30.020/HLS.S30.T10TEK.2021129T184921.v2.0/HLS.S30.T10TEK.2021129T184921.v2.0.jpg']" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#earthdata-cloud-access-information", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#earthdata-cloud-access-information", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Earthdata Cloud access information", + "text": "Earthdata Cloud access information\nClicking the green Download Data button again, will take us to the final page for instructions to download and links for data access in the cloud. The AWS S3 Access tab provides the S3:// links, which is what we would use to access the data directly in-region (us-west-2) within the AWS cloud.\nE.g.: s3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-09_ECCO_V4r4_latlon_0p50deg.nc where s3 indicates data is stored in AWS S3 storage, podaac-ops-cumulus-protected is the bucket, and ECCO_L4_SSH_05DEG_MONTHLY_V4R4 is the object prefix (the latter two are also listed in the dataset collection information under Cloud Access (step 3 above)).\n\nIntegrate file links into programmatic workflow, locally or in the AWS cloud.\nIn the next two examples we will work programmatically in the cloud to access datasets of interest, to get us set up for further scientific analysis of choice. There are several ways to do this. One way to connect the search part of the workflow we just did in Earthdata Search to our next steps working in the cloud is to simply copy/paste the s3:// links provides in Step 4 above into a JupyterHub notebook or script in our cloud workspace, and continue the data analysis from there.\nOne could also copy/paste the s3:// links and save them in a text file, then open and read the text file in the notebook or script in the JupyterHub in the cloud.\n\n\n\nFigure caption: Direct S3 access" }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#accessing-hls-cloud-optimized-geotiffs-cogs-from-earthdata-cloud", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#accessing-hls-cloud-optimized-geotiffs-cogs-from-earthdata-cloud", - "title": "Observing Seasonality in Agricultural Areas", - "section": "3. Accessing HLS Cloud Optimized GeoTIFFs (COGs) from Earthdata Cloud ", - "text": "3. Accessing HLS Cloud Optimized GeoTIFFs (COGs) from Earthdata Cloud \nNow that we have a list of data URLs, we will configure gdal and rioxarray to access the cloud assets that we are interested in, and read them directly into memory without needing to download the files.\nThe Python libraries used to access COG files in Earthdata Cloud leverage GDAL’s virtual file systems. Whether you are running this code in the Cloud or in a local workspace, GDAL configurations must be set in order to successfully access the HLS COG files.\n\n# GDAL configurations used to successfully access LP DAAC Cloud Assets via vsicurl \ngdal.SetConfigOption('GDAL_HTTP_COOKIEFILE','~/cookies.txt')\ngdal.SetConfigOption('GDAL_HTTP_COOKIEJAR', '~/cookies.txt')\ngdal.SetConfigOption('GDAL_DISABLE_READDIR_ON_OPEN','EMPTY_DIR')\ngdal.SetConfigOption('CPL_VSIL_CURL_ALLOWED_EXTENSIONS','TIF')\ngdal.SetConfigOption('GDAL_HTTP_UNSAFESSL', 'YES')" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#part-2-working-with-cloud-optimized-geotiffs-using-nasas-common-metadata-repository-spatio-temporal-assett-catalog-cmr-stac", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#part-2-working-with-cloud-optimized-geotiffs-using-nasas-common-metadata-repository-spatio-temporal-assett-catalog-cmr-stac", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Part 2: Working with Cloud-Optimized GeoTIFFs using NASA’s Common Metadata Repository Spatio-Temporal Assett Catalog (CMR-STAC)", + "text": "Part 2: Working with Cloud-Optimized GeoTIFFs using NASA’s Common Metadata Repository Spatio-Temporal Assett Catalog (CMR-STAC)\nIn this example we will access the NASA’s Harmonized Landsat Sentinel-2 (HLS) version 2 assets, which are archived in cloud optimized geoTIFF (COG) format archived by the Land Processes (LP) DAAC. The COGs can be used like any other GeoTIFF file, but have some added features that make them more efficient within the cloud data access paradigm. These features include: overviews and internal tiling.\n\nBut first, what is STAC?\nSpatioTemporal Asset Catalog (STAC) is a specification that provides a common language for interpreting geospatial information in order to standardize indexing and discovering data.\nThe STAC specification is made up of a collection of related, yet independent specifications that when used together provide search and discovery capabilities for remote assets.\n\nFour STAC Specifications\nSTAC Catalog (aka DAAC Archive)\nSTAC Collection (aka Data Product)\nSTAC Item (aka Granule)\nSTAC API\n\n\n\nCMR-STAC API\nThe CMR-STAC API is NASA’s implementation of the STAC API specification for all NASA data holdings within EOSDIS. The current implementation does not allow for querries accross the entire NASA catalog. Users must execute searches within provider catalogs (e.g., LPCLOUD) to find the STAC Items they are searching for. All the providers can be found at the CMR-STAC endpoint here: https://cmr.earthdata.nasa.gov/stac/.\nIn this example, we will query the LPCLOUD provider to identify STAC Items from the Harmonized Landsat Sentinel-2 (HLS) collection that fall within our region of interest (ROI) and within our specified time range." }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#subset-by-band", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#subset-by-band", - "title": "Observing Seasonality in Agricultural Areas", - "section": "3.1 Subset by Band", - "text": "3.1 Subset by Band\nLets take a look at the URLs for one of our returned granules.\n\nh = hls_results_urls[10] \nh\n\n['https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B12.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B10.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B01.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.Fmask.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B8A.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B09.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B11.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.VAA.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B08.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.SZA.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B07.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B06.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B02.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.VZA.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B03.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B05.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.SAA.tif']\n\n\nTo calculate the EVI for each granule we need the near-infrared, red, and blue bands. Below you can find the different band numbers for each of the two products.\n\nSentinel 2:\n - \"narrow\" NIR = B8A\n - Red = B04\n - Blue = B02 \n - Quality = Fmask\n\n\nLandsat 8:\n - NIR = B05\n - Red = B04\n - Blue = B02 \n - Quality = Fmask\nWe will subset our URLs to include the bands identified above in addition to the quality layer (Fmask).\n\nevi_band_links = []\n\n# Define which HLS product is being accessed\nif h[0].split('/')[4] == 'HLSS30.020':\n evi_bands = ['B8A', 'B04', 'B02', 'Fmask'] # NIR RED BLUE Quality for S30\nelse:\n evi_bands = ['B05', 'B04', 'B02', 'Fmask'] # NIR RED BLUE Quality for L30\n\n# Subset the assets in the item down to only the desired bands\nfor a in h: \n if any(b in a for b in evi_bands):\n evi_band_links.append(a)\nevi_band_links\n\n['https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.Fmask.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B8A.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B02.tif']\n\n\nRemember from above that you can always quickly load in the browse image to get a quick view of the item using our list of browse URLs.\n\nimage = io.imread(browse_urls[10]) # Load jpg browse image into memory\n\n# Basic plot of the image\nplt.figure(figsize=(10,10)) \nplt.imshow(image)\nplt.show()\n\n\n\n\nAbove, we see our first observation over the northern Central Valley of California.\n\ndel image # Remove the browse image" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#import-packages", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#import-packages", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Import packages", + "text": "Import packages\n\nimport os\nimport requests \nimport boto3\nfrom osgeo import gdal\nimport rasterio as rio\nfrom rasterio.session import AWSSession\nimport rioxarray\nimport hvplot.xarray\nimport holoviews as hv\n\nfrom pystac_client import Client \nfrom collections import defaultdict \nimport json\nimport geopandas\nimport geoviews as gv\nfrom cartopy import crs\ngv.extension('bokeh', 'matplotlib')" }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#load-hls-cogs-into-memory", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#load-hls-cogs-into-memory", - "title": "Observing Seasonality in Agricultural Areas", - "section": "3.2 Load HLS COGs into Memory", - "text": "3.2 Load HLS COGs into Memory\nHLS COGs are broken into chunks allowing data to be read more efficiently. Define the chunk size of an HLS tile, mask the NaN values, then read the files using rioxarray and name them based upon the band. We also squeeze the object to remove the band dimension from most of the files, since there is only 1 band.\n\nNOTE: To scale the bands, you can set the mask_and_scale to True (mask_and_scale=True), however the scale_factor information in some of the HLSL30 granules are found in the file metadata, but missing from the Band metadata. rioxarray looks for the scale_factor under Band metadata and if this information is missing it assumes the scale_factor is equal to 1. This results in having data to be uscaled and not masked for those granules. That is why we treat our data a bit differently here, leaving it unscaled and manually updating the scale_factor attribute in the xarray dataarray.\n\n\n# Use vsicurl to load the data directly into memory (be patient, may take a few seconds)\nchunk_size = dict(band=1, x=512, y=512) # Tiles have 1 band and are divided into 512x512 pixel chunks\nfor e in evi_band_links:\n print(e)\n if e.rsplit('.', 2)[-2] == evi_bands[0]: # NIR index\n nir = rxr.open_rasterio(e, chunks=chunk_size, masked=True).squeeze('band', drop=True)\n nir.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute \n elif e.rsplit('.', 2)[-2] == evi_bands[1]: # red index\n red = rxr.open_rasterio(e, chunks=chunk_size, masked=True).squeeze('band', drop=True)\n red.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute\n elif e.rsplit('.', 2)[-2] == evi_bands[2]: # blue index\n blue = rxr.open_rasterio(e, chunks=chunk_size, masked=True).squeeze('band', drop=True)\n blue.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute\n # elif e.rsplit('.', 2)[-2] == evi_bands[3]: # Fmask index\n # fmask = rxr.open_rasterio(e, chunks=chunk_size, masked=True).astype('uint16').squeeze('band', drop=True)\nprint(\"The COGs have been loaded into memory!\")\n\nhttps://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.Fmask.tif\nhttps://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B04.tif\nhttps://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B8A.tif\nhttps://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.B02.tif\nThe COGs have been loaded into memory!\n\n\n\nrxr.open_rasterio('https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T10TEK.2021142T185921.v2.0/HLS.S30.T10TEK.2021142T185921.v2.0.Fmask.tif')\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (band: 1, y: 3660, x: 3660)>\n[13395600 values with dtype=uint8]\nCoordinates:\n * band (band) int32 1\n * x (x) float64 5e+05 5e+05 5.001e+05 ... 6.097e+05 6.098e+05\n * y (y) float64 4.5e+06 4.5e+06 4.5e+06 ... 4.39e+06 4.39e+06\n spatial_ref int32 0\nAttributes: (12/42)\n ACCODE: LaSRC\n AREA_OR_POINT: Area\n arop_ave_xshift(meters): 0\n arop_ave_yshift(meters): 0\n arop_ncp: 0\n arop_rmse(meters): 0\n ... ...\n TILE_ID: S2A_OPER_MSI_L1C_TL_VG...\n ULX: 499980\n ULY: 4500000\n _FillValue: 255\n scale_factor: 1.0\n add_offset: 0.0xarray.DataArrayband: 1y: 3660x: 3660...[13395600 values with dtype=uint8]Coordinates: (4)band(band)int321array([1])x(x)float645e+05 5e+05 ... 6.097e+05 6.098e+05array([499995., 500025., 500055., ..., 609705., 609735., 609765.])y(y)float644.5e+06 4.5e+06 ... 4.39e+06array([4499985., 4499955., 4499925., ..., 4390275., 4390245., 4390215.])spatial_ref()int320crs_wkt :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not specified (based on WGS 84 spheroid)projected_crs_name :UTM Zone 10, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-123.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :499980.0 30.0 0.0 4500000.0 0.0 -30.0array(0)Indexes: (3)bandPandasIndexPandasIndex(Index([1], dtype='int32', name='band'))xPandasIndexPandasIndex(Index([499995.0, 500025.0, 500055.0, 500085.0, 500115.0, 500145.0, 500175.0,\n 500205.0, 500235.0, 500265.0,\n ...\n 609495.0, 609525.0, 609555.0, 609585.0, 609615.0, 609645.0, 609675.0,\n 609705.0, 609735.0, 609765.0],\n dtype='float64', name='x', length=3660))yPandasIndexPandasIndex(Index([4499985.0, 4499955.0, 4499925.0, 4499895.0, 4499865.0, 4499835.0,\n 4499805.0, 4499775.0, 4499745.0, 4499715.0,\n ...\n 4390485.0, 4390455.0, 4390425.0, 4390395.0, 4390365.0, 4390335.0,\n 4390305.0, 4390275.0, 4390245.0, 4390215.0],\n dtype='float64', name='y', length=3660))Attributes: (42)ACCODE :LaSRCAREA_OR_POINT :Areaarop_ave_xshift(meters) :0arop_ave_yshift(meters) :0arop_ncp :0arop_rmse(meters) :0arop_s2_refimg :NONEcloud_coverage :31DATASTRIP_ID :S2A_OPER_MSI_L1C_DS_VGS4_20210523T002123_S20210522T191158_N03.00Fmask description :Bits are listed from the MSB (bit 7) to the LSB (bit 0): \n7-6 aerosol:\n 00 - climatology\n 01 - low\n 10 - average\n 11 - high\n5 water\n4 snow/ice\n3 cloud shadow\n2 adjacent to cloud\n1 cloud\n0 cirrus cloud\nHLS_PROCESSING_TIME :2021-10-24T00:46:20ZHORIZONTAL_CS_CODE :EPSG:32610HORIZONTAL_CS_NAME :WGS84 / UTM zone 10NL1C_IMAGE_QUALITY :NONEL1_PROCESSING_TIME :2021-05-23T00:58:53.432141ZMEAN_SUN_AZIMUTH_ANGLE :146.1715126044MEAN_SUN_ZENITH_ANGLE :22.6352110476638MEAN_VIEW_AZIMUTH_ANGLE :284.987205692531MEAN_VIEW_ZENITH_ANGLE :7.8847569388168MSI band 01 bandpass adjustment slope and offset :0.995900, -0.000200MSI band 02 bandpass adjustment slope and offset :0.977800, -0.004000MSI band 03 bandpass adjustment slope and offset :1.005300, -0.000900MSI band 04 bandpass adjustment slope and offset :0.976500, 0.000900MSI band 11 bandpass adjustment slope and offset :0.998700, -0.001100MSI band 12 bandpass adjustment slope and offset :1.003000, -0.001200MSI band 8a bandpass adjustment slope and offset :0.998300, -0.000100NBAR_SOLAR_ZENITH :24.4031571183093NCOLS :3660NROWS :3660OVR_RESAMPLING_ALG :NEARESTPROCESSING_BASELINE :3.0PRODUCT_URI :S2A_MSIL1C_20210522T185921_N0300_R013_T10TEK_20210523T002123.SAFESENSING_TIME :2021-05-22T19:13:22.446684ZSPACECRAFT_NAME :Sentinel-2Aspatial_coverage :92SPATIAL_RESOLUTION :30TILE_ID :S2A_OPER_MSI_L1C_TL_VGS4_20210523T002123_A030898_T10TEK_N03.00ULX :499980ULY :4500000_FillValue :255scale_factor :1.0add_offset :0.0\n\n\n\nNOTE: Getting an error in the section above? Accessing these files in the cloud requires you to authenticate using your NASA Earthdata Login account. You will need to have a netrc file set up containing those credentials in your home directory in order to successfully run the code below. Please make sure you have a valid username and password in the created netrc file.\n\nWe can take a quick look at one of the dataarray we just read in.\n\nnir\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (y: 3660, x: 3660)>\ndask.array<getitem, shape=(3660, 3660), dtype=float32, chunksize=(512, 512), chunktype=numpy.ndarray>\nCoordinates:\n * x (x) float64 5e+05 5e+05 5.001e+05 ... 6.097e+05 6.098e+05\n * y (y) float64 4.5e+06 4.5e+06 4.5e+06 ... 4.39e+06 4.39e+06\n spatial_ref int32 0\nAttributes: (12/41)\n ACCODE: LaSRC\n add_offset: 0.0\n AREA_OR_POINT: Area\n arop_ave_xshift(meters): 0\n arop_ave_yshift(meters): 0\n arop_ncp: 0\n ... ...\n SPACECRAFT_NAME: Sentinel-2A\n spatial_coverage: 92\n SPATIAL_RESOLUTION: 30\n TILE_ID: S2A_OPER_MSI_L1C_TL_VG...\n ULX: 499980\n ULY: 4500000xarray.DataArrayy: 3660x: 3660dask.array<chunksize=(512, 512), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n51.10 MiB\n1.00 MiB\n\n\nShape\n(3660, 3660)\n(512, 512)\n\n\nDask graph\n64 chunks in 3 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\nCoordinates: (3)x(x)float645e+05 5e+05 ... 6.097e+05 6.098e+05array([499995., 500025., 500055., ..., 609705., 609735., 609765.])y(y)float644.5e+06 4.5e+06 ... 4.39e+06array([4499985., 4499955., 4499925., ..., 4390275., 4390245., 4390215.])spatial_ref()int320crs_wkt :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not specified (based on WGS 84 spheroid)projected_crs_name :UTM Zone 10, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-123.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :499980.0 30.0 0.0 4500000.0 0.0 -30.0array(0)Indexes: (2)xPandasIndexPandasIndex(Index([499995.0, 500025.0, 500055.0, 500085.0, 500115.0, 500145.0, 500175.0,\n 500205.0, 500235.0, 500265.0,\n ...\n 609495.0, 609525.0, 609555.0, 609585.0, 609615.0, 609645.0, 609675.0,\n 609705.0, 609735.0, 609765.0],\n dtype='float64', name='x', length=3660))yPandasIndexPandasIndex(Index([4499985.0, 4499955.0, 4499925.0, 4499895.0, 4499865.0, 4499835.0,\n 4499805.0, 4499775.0, 4499745.0, 4499715.0,\n ...\n 4390485.0, 4390455.0, 4390425.0, 4390395.0, 4390365.0, 4390335.0,\n 4390305.0, 4390275.0, 4390245.0, 4390215.0],\n dtype='float64', name='y', length=3660))Attributes: (41)ACCODE :LaSRCadd_offset :0.0AREA_OR_POINT :Areaarop_ave_xshift(meters) :0arop_ave_yshift(meters) :0arop_ncp :0arop_rmse(meters) :0arop_s2_refimg :NONEcloud_coverage :31DATASTRIP_ID :S2A_OPER_MSI_L1C_DS_VGS4_20210523T002123_S20210522T191158_N03.00HLS_PROCESSING_TIME :2021-10-24T00:46:20ZHORIZONTAL_CS_CODE :EPSG:32610HORIZONTAL_CS_NAME :WGS84 / UTM zone 10NL1C_IMAGE_QUALITY :NONEL1_PROCESSING_TIME :2021-05-23T00:58:53.432141Zlong_name :NIR_NarrowMEAN_SUN_AZIMUTH_ANGLE :146.1715126044MEAN_SUN_ZENITH_ANGLE :22.6352110476638MEAN_VIEW_AZIMUTH_ANGLE :284.987205692531MEAN_VIEW_ZENITH_ANGLE :7.8847569388168MSI band 01 bandpass adjustment slope and offset :0.995900, -0.000200MSI band 02 bandpass adjustment slope and offset :0.977800, -0.004000MSI band 03 bandpass adjustment slope and offset :1.005300, -0.000900MSI band 04 bandpass adjustment slope and offset :0.976500, 0.000900MSI band 11 bandpass adjustment slope and offset :0.998700, -0.001100MSI band 12 bandpass adjustment slope and offset :1.003000, -0.001200MSI band 8a bandpass adjustment slope and offset :0.998300, -0.000100NBAR_SOLAR_ZENITH :24.4031571183093NCOLS :3660NROWS :3660OVR_RESAMPLING_ALG :NEARESTPROCESSING_BASELINE :3.0PRODUCT_URI :S2A_MSIL1C_20210522T185921_N0300_R013_T10TEK_20210523T002123.SAFEscale_factor :0.0001SENSING_TIME :2021-05-22T19:13:22.446684ZSPACECRAFT_NAME :Sentinel-2Aspatial_coverage :92SPATIAL_RESOLUTION :30TILE_ID :S2A_OPER_MSI_L1C_TL_VGS4_20210523T002123_A030898_T10TEK_N03.00ULX :499980ULY :4500000\n\n\nNote the full size of the array, y=3660 & x=3660" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#connect-to-the-cmr-stac-api", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#connect-to-the-cmr-stac-api", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Connect to the CMR-STAC API", + "text": "Connect to the CMR-STAC API\n\nSTAC_URL = 'https://cmr.earthdata.nasa.gov/stac'\n\n\nprovider_cat = Client.open(STAC_URL)\n\n\nConnect to the LPCLOUD Provider/STAC Catalog\nFor this next step we need the provider title (e.g., LPCLOUD). We will add the provider to the end of the CMR-STAC API URL (i.e., https://cmr.earthdata.nasa.gov/stac/) to connect to the LPCLOUD STAC Catalog.\n\ncatalog = Client.open(f'{STAC_URL}/LPCLOUD/')\n\nSince we are using a dedicated client (i.e., pystac-client.Client) to connect to our STAC Provider Catalog, we will have access to some useful internal methods and functions (e.g., get_children() or get_all_items()) we can use to get information from these objects." }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#subset-spatially", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#subset-spatially", - "title": "Observing Seasonality in Agricultural Areas", - "section": "3.3 Subset spatially", - "text": "3.3 Subset spatially\nBefore we can subset using our input farm field, we will first need to convert the geopandas dataframe from lat/lon (EPSG: 4326) into the projection used by HLS, UTM (aligned to the Military Grid Reference System). Since UTM is a zonal projection, we’ll need to extract the unique UTM zonal projection parameters from our input HLS files and use them to transform the coordinate of our input farm field.\nWe can print out the WKT string for our HLS tiles.\n\nnir.spatial_ref.crs_wkt\n\n'PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]'\n\n\nWe will use this information to transform the coordinates of our ROI to the proper UTM projection.\n\nfsUTM = field.to_crs(nir.spatial_ref.crs_wkt) # Take the CRS from the NIR tile that we opened and apply it to our field geodataframe.\nfsUTM\n\n\n\n\n\n\n\n\ngeometry\n\n\n\n\n0\nPOLYGON ((581047.926 4418541.913, 580150.196 4...\n\n\n\n\n\n\n\nNow, we can use our field ROI to mask any pixels that fall outside of it and crop to the bounding box using rasterio. This greatly reduces the amount of data that are needed to load into memory.\n\nnir_cropped = nir.rio.clip(fsUTM.geometry.values, fsUTM.crs, all_touched=True) # All touched includes any pixels touched by the polygon\nnir_cropped\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (y: 59, x: 38)>\ndask.array<getitem, shape=(59, 38), dtype=float32, chunksize=(59, 38), chunktype=numpy.ndarray>\nCoordinates:\n * x (x) float64 5.802e+05 5.802e+05 ... 5.812e+05 5.813e+05\n * y (y) float64 4.419e+06 4.419e+06 ... 4.417e+06 4.417e+06\n spatial_ref int32 0\nAttributes: (12/41)\n ACCODE: LaSRC\n add_offset: 0.0\n AREA_OR_POINT: Area\n arop_ave_xshift(meters): 0\n arop_ave_yshift(meters): 0\n arop_ncp: 0\n ... ...\n SPACECRAFT_NAME: Sentinel-2A\n spatial_coverage: 92\n SPATIAL_RESOLUTION: 30\n TILE_ID: S2A_OPER_MSI_L1C_TL_VG...\n ULX: 499980\n ULY: 4500000xarray.DataArrayy: 59x: 38dask.array<chunksize=(59, 38), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n8.76 kiB\n8.76 kiB\n\n\nShape\n(59, 38)\n(59, 38)\n\n\nDask graph\n1 chunks in 7 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\nCoordinates: (3)x(x)float645.802e+05 5.802e+05 ... 5.813e+05axis :Xlong_name :x coordinate of projectionstandard_name :projection_x_coordinateunits :metrearray([580155., 580185., 580215., 580245., 580275., 580305., 580335., 580365.,\n 580395., 580425., 580455., 580485., 580515., 580545., 580575., 580605.,\n 580635., 580665., 580695., 580725., 580755., 580785., 580815., 580845.,\n 580875., 580905., 580935., 580965., 580995., 581025., 581055., 581085.,\n 581115., 581145., 581175., 581205., 581235., 581265.])y(y)float644.419e+06 4.419e+06 ... 4.417e+06axis :Ylong_name :y coordinate of projectionstandard_name :projection_y_coordinateunits :metrearray([4418535., 4418505., 4418475., 4418445., 4418415., 4418385., 4418355.,\n 4418325., 4418295., 4418265., 4418235., 4418205., 4418175., 4418145.,\n 4418115., 4418085., 4418055., 4418025., 4417995., 4417965., 4417935.,\n 4417905., 4417875., 4417845., 4417815., 4417785., 4417755., 4417725.,\n 4417695., 4417665., 4417635., 4417605., 4417575., 4417545., 4417515.,\n 4417485., 4417455., 4417425., 4417395., 4417365., 4417335., 4417305.,\n 4417275., 4417245., 4417215., 4417185., 4417155., 4417125., 4417095.,\n 4417065., 4417035., 4417005., 4416975., 4416945., 4416915., 4416885.,\n 4416855., 4416825., 4416795.])spatial_ref()int320crs_wkt :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not specified (based on WGS 84 spheroid)projected_crs_name :UTM Zone 10, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-123.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :580140.0 30.0 0.0 4418550.0 0.0 -30.0array(0)Indexes: (2)xPandasIndexPandasIndex(Index([580155.0, 580185.0, 580215.0, 580245.0, 580275.0, 580305.0, 580335.0,\n 580365.0, 580395.0, 580425.0, 580455.0, 580485.0, 580515.0, 580545.0,\n 580575.0, 580605.0, 580635.0, 580665.0, 580695.0, 580725.0, 580755.0,\n 580785.0, 580815.0, 580845.0, 580875.0, 580905.0, 580935.0, 580965.0,\n 580995.0, 581025.0, 581055.0, 581085.0, 581115.0, 581145.0, 581175.0,\n 581205.0, 581235.0, 581265.0],\n dtype='float64', name='x'))yPandasIndexPandasIndex(Index([4418535.0, 4418505.0, 4418475.0, 4418445.0, 4418415.0, 4418385.0,\n 4418355.0, 4418325.0, 4418295.0, 4418265.0, 4418235.0, 4418205.0,\n 4418175.0, 4418145.0, 4418115.0, 4418085.0, 4418055.0, 4418025.0,\n 4417995.0, 4417965.0, 4417935.0, 4417905.0, 4417875.0, 4417845.0,\n 4417815.0, 4417785.0, 4417755.0, 4417725.0, 4417695.0, 4417665.0,\n 4417635.0, 4417605.0, 4417575.0, 4417545.0, 4417515.0, 4417485.0,\n 4417455.0, 4417425.0, 4417395.0, 4417365.0, 4417335.0, 4417305.0,\n 4417275.0, 4417245.0, 4417215.0, 4417185.0, 4417155.0, 4417125.0,\n 4417095.0, 4417065.0, 4417035.0, 4417005.0, 4416975.0, 4416945.0,\n 4416915.0, 4416885.0, 4416855.0, 4416825.0, 4416795.0],\n dtype='float64', name='y'))Attributes: (41)ACCODE :LaSRCadd_offset :0.0AREA_OR_POINT :Areaarop_ave_xshift(meters) :0arop_ave_yshift(meters) :0arop_ncp :0arop_rmse(meters) :0arop_s2_refimg :NONEcloud_coverage :31DATASTRIP_ID :S2A_OPER_MSI_L1C_DS_VGS4_20210523T002123_S20210522T191158_N03.00HLS_PROCESSING_TIME :2021-10-24T00:46:20ZHORIZONTAL_CS_CODE :EPSG:32610HORIZONTAL_CS_NAME :WGS84 / UTM zone 10NL1C_IMAGE_QUALITY :NONEL1_PROCESSING_TIME :2021-05-23T00:58:53.432141Zlong_name :NIR_NarrowMEAN_SUN_AZIMUTH_ANGLE :146.1715126044MEAN_SUN_ZENITH_ANGLE :22.6352110476638MEAN_VIEW_AZIMUTH_ANGLE :284.987205692531MEAN_VIEW_ZENITH_ANGLE :7.8847569388168MSI band 01 bandpass adjustment slope and offset :0.995900, -0.000200MSI band 02 bandpass adjustment slope and offset :0.977800, -0.004000MSI band 03 bandpass adjustment slope and offset :1.005300, -0.000900MSI band 04 bandpass adjustment slope and offset :0.976500, 0.000900MSI band 11 bandpass adjustment slope and offset :0.998700, -0.001100MSI band 12 bandpass adjustment slope and offset :1.003000, -0.001200MSI band 8a bandpass adjustment slope and offset :0.998300, -0.000100NBAR_SOLAR_ZENITH :24.4031571183093NCOLS :3660NROWS :3660OVR_RESAMPLING_ALG :NEARESTPROCESSING_BASELINE :3.0PRODUCT_URI :S2A_MSIL1C_20210522T185921_N0300_R013_T10TEK_20210523T002123.SAFEscale_factor :0.0001SENSING_TIME :2021-05-22T19:13:22.446684ZSPACECRAFT_NAME :Sentinel-2Aspatial_coverage :92SPATIAL_RESOLUTION :30TILE_ID :S2A_OPER_MSI_L1C_TL_VGS4_20210523T002123_A030898_T10TEK_N03.00ULX :499980ULY :4500000\n\n\nNote that the array size is considerably smaller than the full size we read in before.\nNow we will plot the cropped NIR data.\n\nnir_cropped.hvplot.image(aspect='equal', cmap='viridis', frame_height=500, frame_width= 500).opts(title='HLS Cropped NIR Band') # Quick visual to assure that it worked\n\n\n\n\n\n \n\n\n\n\nAbove, you can see that the data have been loaded into memory already subset to our ROI. Also notice that the data has not been scaled (see the legend). We will next scaled the data using the function defined below." + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#search-for-stac-items", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#search-for-stac-items", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Search for STAC Items", + "text": "Search for STAC Items\nWe will define our ROI using a geojson file containing a small polygon feature in western Nebraska, USA. We’ll also specify the data collections and a time range for our example.\n\nRead in a geojson file and plot\nReading in a geojson file with geopandas and extract coodinates for our ROI.\n\nfield = geopandas.read_file('../data/ne_w_agfields.geojson')\nfieldShape = field['geometry'][0]\nroi = json.loads(field.to_json())['features'][0]['geometry']\n\nWe can plot the polygon using the geoviews package that we imported as gv with ‘bokeh’ and ‘matplotlib’ extensions. The following has reasonable width, height, color, and line widths to view our polygon when it is overlayed on a base tile map.\n\nbase = gv.tile_sources.EsriImagery.opts(width=650, height=500)\nfarmField = gv.Polygons(fieldShape).opts(line_color='yellow', line_width=10, color=None)\nbase * farmField\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\nWe will now start to specify the search criteria we are interested in, i.e, the date range, the ROI, and the data collections, that we will pass to the STAC API." }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#apply-scale-factor", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#apply-scale-factor", - "title": "Observing Seasonality in Agricultural Areas", - "section": "3.4 Apply Scale Factor", - "text": "3.4 Apply Scale Factor\n\n# Define function to scale \ndef scaling(band):\n scale_factor = band.attrs['scale_factor'] \n band_out = band.copy()\n band_out.data = band.data*scale_factor\n band_out.attrs['scale_factor'] = 1\n return(band_out)\n\n\nnir_cropped_scaled = scaling(nir_cropped)\n\nWe can plot to confirm our manual scaling worked.\n\nnir_cropped_scaled.hvplot.image(aspect='equal', cmap='viridis', frame_height=500, frame_width= 500).opts(title='HLS Cropped NIR Band') # Quick visual to assure that it worked\n\n\n\n\n\n \n\n\n\n\nNext, load in the red and blue bands and fix their scaling as well.\n\n# Red\nred_cropped = red.rio.clip(fsUTM.geometry.values, fsUTM.crs, all_touched=True)\nred_cropped_scaled = scaling(red_cropped)\n# Blue\nblue_cropped = blue.rio.clip(fsUTM.geometry.values, fsUTM.crs, all_touched=True)\nblue_cropped_scaled = scaling(blue_cropped)\nprint('Data is loaded and scaled!')\n\nData is loaded and scaled!" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#search-the-cmr-stac-api-with-our-search-criteria", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#search-the-cmr-stac-api-with-our-search-criteria", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Search the CMR-STAC API with our search criteria", + "text": "Search the CMR-STAC API with our search criteria\nNow we can put all our search criteria together using catalog.search from the pystac_client package. STAC Collection is synonomous with what we usually consider a NASA data product. Desired STAC Collections are submitted to the search API as a list containing the collection id. Let’s focus on S30 and L30 collections.\n\ncollections = ['HLSL30.v2.0', 'HLSS30.v2.0']\n\ndate_range = \"2021-05/2021-08\"\n\nsearch = catalog.search(\n collections=collections,\n intersects=roi,\n datetime=date_range,\n limit=100\n)\n\n\nView STAC Items that matched our search query\n\nprint('Matching STAC Items:', search.matched())\nitem_collection = search.get_all_items()\nitem_collection[0].to_dict()\n\nMatching STAC Items: 113\n\n\n{'type': 'Feature',\n 'stac_version': '1.0.0',\n 'id': 'HLS.L30.T13TGF.2021124T173013.v2.0',\n 'properties': {'datetime': '2021-05-04T17:30:13.428000Z',\n 'start_datetime': '2021-05-04T17:30:13.428Z',\n 'end_datetime': '2021-05-04T17:30:37.319Z',\n 'eo:cloud_cover': 36},\n 'geometry': {'type': 'Polygon',\n 'coordinates': [[[-101.5423534, 40.5109845],\n [-101.3056118, 41.2066375],\n [-101.2894253, 41.4919436],\n [-102.6032964, 41.5268623],\n [-102.638891, 40.5386175],\n [-101.5423534, 40.5109845]]]},\n 'links': [{'rel': 'self',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0/items/HLS.L30.T13TGF.2021124T173013.v2.0'},\n {'rel': 'parent',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0'},\n {'rel': 'collection',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0'},\n {'rel': <RelType.ROOT: 'root'>,\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/',\n 'type': <MediaType.JSON: 'application/json'>,\n 'title': 'LPCLOUD'},\n {'rel': 'provider', 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD'},\n {'rel': 'via',\n 'href': 'https://cmr.earthdata.nasa.gov/search/concepts/G2144020713-LPCLOUD.json'},\n {'rel': 'via',\n 'href': 'https://cmr.earthdata.nasa.gov/search/concepts/G2144020713-LPCLOUD.umm_json'}],\n 'assets': {'B11': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B11.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B11.tif'},\n 'B07': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B07.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B07.tif'},\n 'SAA': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.SAA.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.SAA.tif'},\n 'B06': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B06.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B06.tif'},\n 'B09': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B09.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B09.tif'},\n 'B10': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B10.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B10.tif'},\n 'VZA': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.VZA.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.VZA.tif'},\n 'SZA': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.SZA.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.SZA.tif'},\n 'B01': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B01.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B01.tif'},\n 'VAA': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.VAA.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.VAA.tif'},\n 'B05': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B05.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B05.tif'},\n 'B02': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B02.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B02.tif'},\n 'Fmask': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.Fmask.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.Fmask.tif'},\n 'B03': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B03.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B03.tif'},\n 'B04': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B04.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B04.tif'},\n 'browse': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.jpg',\n 'type': 'image/jpeg',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.jpg'},\n 'metadata': {'href': 'https://cmr.earthdata.nasa.gov/search/concepts/G2144020713-LPCLOUD.xml',\n 'type': 'application/xml'}},\n 'bbox': [-102.638891, 40.510984, -101.289425, 41.526862],\n 'stac_extensions': ['https://stac-extensions.github.io/eo/v1.0.0/schema.json'],\n 'collection': 'HLSL30.v2.0'}" }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#processing-hls-data", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#processing-hls-data", - "title": "Observing Seasonality in Agricultural Areas", - "section": "4. Processing HLS Data ", - "text": "4. Processing HLS Data \nIn this section we will use the HLS data we’ve access to calculate the EVI. We will do this by defining a function to calculate EVI that will retain the attributes and metadata associated with the data we accessed.\n\n4.1 Calculate EVI\nBelow is a function we will use to calculate EVI using the NIR, Red, and Blue bands. The function will build a new xarray dataarray with EVI values and copy the original file metadata to the new xarray dataarray\n\ndef calc_evi(red, blue, nir):\n # Create EVI xarray.DataArray that has the same coordinates and metadata \n evi = red.copy()\n # Calculate the EVI\n evi_data = 2.5 * ((nir.data - red.data) / (nir.data + 6.0 * red.data - 7.5 * blue.data + 1.0))\n # Replace the Red xarray.DataArray data with the new EVI data\n evi.data = evi_data\n # exclude the inf values\n evi = xr.where(evi != np.inf, evi, np.nan, keep_attrs=True)\n # change the long_name in the attributes\n evi.attrs['long_name'] = 'EVI'\n evi.attrs['scale_factor'] = 1\n return evi\n\nBelow, apply the EVI function on the scaled data.\n\nevi_cropped = calc_evi(red_cropped_scaled, blue_cropped_scaled, nir_cropped_scaled) # Generate EVI array\nevi_cropped\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (y: 59, x: 38)>\ndask.array<where, shape=(59, 38), dtype=float32, chunksize=(59, 38), chunktype=numpy.ndarray>\nCoordinates:\n * x (x) float64 5.802e+05 5.802e+05 ... 5.812e+05 5.813e+05\n * y (y) float64 4.419e+06 4.419e+06 ... 4.417e+06 4.417e+06\n spatial_ref int32 0\nAttributes: (12/41)\n ACCODE: LaSRC\n add_offset: 0.0\n AREA_OR_POINT: Area\n arop_ave_xshift(meters): 0\n arop_ave_yshift(meters): 0\n arop_ncp: 0\n ... ...\n SPACECRAFT_NAME: Sentinel-2A\n spatial_coverage: 92\n SPATIAL_RESOLUTION: 30\n TILE_ID: S2A_OPER_MSI_L1C_TL_VG...\n ULX: 499980\n ULY: 4500000xarray.DataArrayy: 59x: 38dask.array<chunksize=(59, 38), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n8.76 kiB\n8.76 kiB\n\n\nShape\n(59, 38)\n(59, 38)\n\n\nDask graph\n1 chunks in 30 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\nCoordinates: (3)x(x)float645.802e+05 5.802e+05 ... 5.813e+05axis :Xlong_name :x coordinate of projectionstandard_name :projection_x_coordinateunits :metrearray([580155., 580185., 580215., 580245., 580275., 580305., 580335., 580365.,\n 580395., 580425., 580455., 580485., 580515., 580545., 580575., 580605.,\n 580635., 580665., 580695., 580725., 580755., 580785., 580815., 580845.,\n 580875., 580905., 580935., 580965., 580995., 581025., 581055., 581085.,\n 581115., 581145., 581175., 581205., 581235., 581265.])y(y)float644.419e+06 4.419e+06 ... 4.417e+06axis :Ylong_name :y coordinate of projectionstandard_name :projection_y_coordinateunits :metrearray([4418535., 4418505., 4418475., 4418445., 4418415., 4418385., 4418355.,\n 4418325., 4418295., 4418265., 4418235., 4418205., 4418175., 4418145.,\n 4418115., 4418085., 4418055., 4418025., 4417995., 4417965., 4417935.,\n 4417905., 4417875., 4417845., 4417815., 4417785., 4417755., 4417725.,\n 4417695., 4417665., 4417635., 4417605., 4417575., 4417545., 4417515.,\n 4417485., 4417455., 4417425., 4417395., 4417365., 4417335., 4417305.,\n 4417275., 4417245., 4417215., 4417185., 4417155., 4417125., 4417095.,\n 4417065., 4417035., 4417005., 4416975., 4416945., 4416915., 4416885.,\n 4416855., 4416825., 4416795.])spatial_ref()int320crs_wkt :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not specified (based on WGS 84 spheroid)projected_crs_name :UTM Zone 10, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-123.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :580140.0 30.0 0.0 4418550.0 0.0 -30.0array(0)Indexes: (2)xPandasIndexPandasIndex(Index([580155.0, 580185.0, 580215.0, 580245.0, 580275.0, 580305.0, 580335.0,\n 580365.0, 580395.0, 580425.0, 580455.0, 580485.0, 580515.0, 580545.0,\n 580575.0, 580605.0, 580635.0, 580665.0, 580695.0, 580725.0, 580755.0,\n 580785.0, 580815.0, 580845.0, 580875.0, 580905.0, 580935.0, 580965.0,\n 580995.0, 581025.0, 581055.0, 581085.0, 581115.0, 581145.0, 581175.0,\n 581205.0, 581235.0, 581265.0],\n dtype='float64', name='x'))yPandasIndexPandasIndex(Index([4418535.0, 4418505.0, 4418475.0, 4418445.0, 4418415.0, 4418385.0,\n 4418355.0, 4418325.0, 4418295.0, 4418265.0, 4418235.0, 4418205.0,\n 4418175.0, 4418145.0, 4418115.0, 4418085.0, 4418055.0, 4418025.0,\n 4417995.0, 4417965.0, 4417935.0, 4417905.0, 4417875.0, 4417845.0,\n 4417815.0, 4417785.0, 4417755.0, 4417725.0, 4417695.0, 4417665.0,\n 4417635.0, 4417605.0, 4417575.0, 4417545.0, 4417515.0, 4417485.0,\n 4417455.0, 4417425.0, 4417395.0, 4417365.0, 4417335.0, 4417305.0,\n 4417275.0, 4417245.0, 4417215.0, 4417185.0, 4417155.0, 4417125.0,\n 4417095.0, 4417065.0, 4417035.0, 4417005.0, 4416975.0, 4416945.0,\n 4416915.0, 4416885.0, 4416855.0, 4416825.0, 4416795.0],\n dtype='float64', name='y'))Attributes: (41)ACCODE :LaSRCadd_offset :0.0AREA_OR_POINT :Areaarop_ave_xshift(meters) :0arop_ave_yshift(meters) :0arop_ncp :0arop_rmse(meters) :0arop_s2_refimg :NONEcloud_coverage :31DATASTRIP_ID :S2A_OPER_MSI_L1C_DS_VGS4_20210523T002123_S20210522T191158_N03.00HLS_PROCESSING_TIME :2021-10-24T00:46:20ZHORIZONTAL_CS_CODE :EPSG:32610HORIZONTAL_CS_NAME :WGS84 / UTM zone 10NL1C_IMAGE_QUALITY :NONEL1_PROCESSING_TIME :2021-05-23T00:58:53.432141Zlong_name :EVIMEAN_SUN_AZIMUTH_ANGLE :146.1715126044MEAN_SUN_ZENITH_ANGLE :22.6352110476638MEAN_VIEW_AZIMUTH_ANGLE :284.987205692531MEAN_VIEW_ZENITH_ANGLE :7.8847569388168MSI band 01 bandpass adjustment slope and offset :0.995900, -0.000200MSI band 02 bandpass adjustment slope and offset :0.977800, -0.004000MSI band 03 bandpass adjustment slope and offset :1.005300, -0.000900MSI band 04 bandpass adjustment slope and offset :0.976500, 0.000900MSI band 11 bandpass adjustment slope and offset :0.998700, -0.001100MSI band 12 bandpass adjustment slope and offset :1.003000, -0.001200MSI band 8a bandpass adjustment slope and offset :0.998300, -0.000100NBAR_SOLAR_ZENITH :24.4031571183093NCOLS :3660NROWS :3660OVR_RESAMPLING_ALG :NEARESTPROCESSING_BASELINE :3.0PRODUCT_URI :S2A_MSIL1C_20210522T185921_N0300_R013_T10TEK_20210523T002123.SAFEscale_factor :1SENSING_TIME :2021-05-22T19:13:22.446684ZSPACECRAFT_NAME :Sentinel-2Aspatial_coverage :92SPATIAL_RESOLUTION :30TILE_ID :S2A_OPER_MSI_L1C_TL_VGS4_20210523T002123_A030898_T10TEK_N03.00ULX :499980ULY :4500000\n\n\nNext, plot the results using hvplot.\n\nevi_cropped.hvplot.image(aspect='equal', cmap='YlGn', frame_height=500, frame_width= 500).opts(title=f'HLS-derived EVI, {evi_cropped.SENSING_TIME}', clabel='EVI')\n\n\n\n\n\n \n\n\n\n\nAbove, notice that variation of green level appearing in different fields in our ROI, some being much greener than the others.\n\n\n4.2. Export to COG\nIn this section, create an output filename and export the cropped EVI to COG.\n\noriginal_name = evi_band_links[0].split('/')[-1]\noriginal_name\n\n'HLS.S30.T10TEK.2021142T185921.v2.0.Fmask.tif'\n\n\nThe standard format for HLS S30 V2.0 and HLS L30 V2.0 filenames is as follows: > HLS.S30/HLS.L30: Product Short Name\nT10TEK: MGRS Tile ID (T+5-digits)\n2020273T190109: Julian Date and Time of Acquisition (YYYYDDDTHHMMSS)\nv2.0: Product Version\nB8A/B05: Spectral Band\n.tif: Data Format (Cloud Optimized GeoTIFF)\nFor additional information on HLS naming conventions, be sure to check out the HLS Overview Page.\nNow modify the filename to describe that its EVI, cropped to an ROI.\n\nout_name = f\"{original_name.split('v2.0')[0]}v2.0_EVI_cropped.tif\" # Generate output name from the original filename\nout_name\n\n'HLS.S30.T10TEK.2021142T185921.v2.0_EVI_cropped.tif'\n\n\nUse the COG driver to write a local raster output. A cloud-optimized geotiff (COG) is a geotiff file that has been tiled and includes overviews so it can be accessed and previewed without having to load the entire image into memory at once.\n\nout_folder = './data/'\nif not os.path.exists(out_folder): os.makedirs(out_folder)\n\nevi_cropped.rio.to_raster(raster_path=f'{out_folder}{out_name}', driver='COG')\n\n\n# del evi_cropped, out_folder, out_name, fmask_cropped, red_cropped, blue_cropped, nir_cropped, red_cropped_scaled, blue_cropped_scaled, nir_cropped_scaled" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#filtering-stac-items", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#filtering-stac-items", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Filtering STAC Items", + "text": "Filtering STAC Items\nBelow we will loop through and filter the item_collection by a specified cloud cover as well as extract the band we’d need to do an Enhanced Vegetation Index (EVI) calculation for a future analysis. We will also specify the STAC Assets (i.e., bands/layers) of interest for both the S30 and L30 collections (also in our collections variable above) and print out the first ten links, converted to s3 locations:\n\ncloudcover = 25\n\ns30_bands = ['B8A', 'B04', 'B02', 'Fmask'] # S30 bands for EVI calculation and quality filtering -> NIR, RED, BLUE, Quality \nl30_bands = ['B05', 'B04', 'B02', 'Fmask'] # L30 bands for EVI calculation and quality filtering -> NIR, RED, BLUE, Quality \n\nevi_band_links = []\n\nfor i in item_collection:\n if i.properties['eo:cloud_cover'] <= cloudcover:\n if i.collection_id == 'HLSS30.v2.0':\n #print(i.properties['eo:cloud_cover'])\n evi_bands = s30_bands\n elif i.collection_id == 'HLSL30.v2.0':\n #print(i.properties['eo:cloud_cover'])\n evi_bands = l30_bands\n\n for a in i.assets:\n if any(b==a for b in evi_bands):\n evi_band_links.append(i.assets[a].href)\n \ns3_links = [l.replace('https://data.lpdaac.earthdatacloud.nasa.gov/', 's3://') for l in evi_band_links]\ns3_links[:10]\n\n['s3://lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B04.tif',\n 's3://lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B05.tif',\n 's3://lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.Fmask.tif',\n 's3://lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B02.tif',\n 's3://lp-prod-protected/HLSL30.020/HLS.L30.T14TKL.2021133T172406.v2.0/HLS.L30.T14TKL.2021133T172406.v2.0.B02.tif',\n 's3://lp-prod-protected/HLSL30.020/HLS.L30.T14TKL.2021133T172406.v2.0/HLS.L30.T14TKL.2021133T172406.v2.0.B04.tif',\n 's3://lp-prod-protected/HLSL30.020/HLS.L30.T14TKL.2021133T172406.v2.0/HLS.L30.T14TKL.2021133T172406.v2.0.B05.tif',\n 's3://lp-prod-protected/HLSL30.020/HLS.L30.T14TKL.2021133T172406.v2.0/HLS.L30.T14TKL.2021133T172406.v2.0.Fmask.tif',\n 's3://lp-prod-protected/HLSS30.020/HLS.S30.T14TKL.2021133T173859.v2.0/HLS.S30.T14TKL.2021133T173859.v2.0.B04.tif',\n 's3://lp-prod-protected/HLSS30.020/HLS.S30.T14TKL.2021133T173859.v2.0/HLS.S30.T14TKL.2021133T173859.v2.0.B8A.tif']" }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#automation", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#automation", - "title": "Observing Seasonality in Agricultural Areas", - "section": "5. Automation ", - "text": "5. Automation \nIn this section, automate sections 4-5 for each HLS item that intersects our spatiotemporal subset of interest. Loop through each item and subset to the desired bands, load the spatial subset into memory, apply the scale factor, calculate EVI, and export as a Cloud Optimized GeoTIFF.\n\nlen(hls_results_urls)\n\n78\n\n\nNow put it all together and loop through each of the files, visualize and export cropped EVI files. Be patient with the for loop below, it may take a few minutes to complete.\n\nfor j, h in enumerate(hls_results_urls):\n \n outName = h[0].split('/')[-1].split('v2.0')[0] +'v2.0_EVI_cropped.tif'\n #print(outName)\n\n evi_band_links = []\n if h[0].split('/')[4] == 'HLSS30.020':\n evi_bands = ['B8A', 'B04', 'B02', 'Fmask'] # NIR RED BLUE FMASK\n else:\n evi_bands = ['B05', 'B04', 'B02', 'Fmask'] # NIR RED BLUE FMASK\n \n for a in h: \n if any(b in a for b in evi_bands):\n evi_band_links.append(a)\n\n \n # Check if file already exists in output directory, if yes--skip that file and move to the next observation\n if os.path.exists(f'{out_folder}{outName}'):\n print(f\"{outName} has already been processed and is available in this directory, moving to next file.\")\n continue\n \n # Use vsicurl to load the data directly into memory (be patient, may take a few seconds)\n chunk_size = dict(band=1, x=512, y=512) # Tiles have 1 band and are divided into 512x512 pixel chunks\n for e in evi_band_links:\n #print(e)\n if e.rsplit('.', 2)[-2] == evi_bands[0]: # NIR index\n nir = rxr.open_rasterio(e, chunks=chunk_size, masked= True).squeeze('band', drop=True)\n nir.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute \n elif e.rsplit('.', 2)[-2] == evi_bands[1]: # red index\n red = rxr.open_rasterio(e, chunks=chunk_size, masked= True).squeeze('band', drop=True)\n red.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute\n elif e.rsplit('.', 2)[-2] == evi_bands[2]: # blue index\n blue = rxr.open_rasterio(e, chunks=chunk_size, masked= True).squeeze('band', drop=True)\n blue.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute\n elif e.rsplit('.', 2)[-2] == evi_bands[3]: # Fmask index\n fmask = rxr.open_rasterio(e, chunks=chunk_size, masked= True).astype('uint16').squeeze('band', drop=True)\n\n \n #print(\"The COGs have been loaded into memory!\")\n \n fsUTM = field.to_crs(nir.spatial_ref.crs_wkt)\n\n # Crop to our ROI and apply scaling and masking\n nir_cropped = nir.rio.clip(fsUTM.geometry.values, fsUTM.crs, all_touched=True)\n red_cropped = red.rio.clip(fsUTM.geometry.values, fsUTM.crs, all_touched=True)\n blue_cropped = blue.rio.clip(fsUTM.geometry.values, fsUTM.crs, all_touched=True)\n fmask_cropped = fmask.rio.clip(fsUTM.geometry.values, fsUTM.crs, all_touched=True)\n \n #print('Cropped') \n \n # Fix Scaling\n nir_cropped_scaled = scaling(nir_cropped)\n red_cropped_scaled = scaling(red_cropped)\n blue_cropped_scaled = scaling(blue_cropped)\n\n\n # Generate EVI\n \n evi_cropped = calc_evi(red_cropped_scaled, blue_cropped_scaled, nir_cropped_scaled)\n\n #print('EVI Calculated')\n \n # Remove any observations that are entirely fill value\n if np.nansum(evi_cropped.data) == 0.0:\n print(f\"File: {h[0].split('/')[-1].rsplit('.', 1)[0]} was entirely fill values and will not be exported.\")\n continue\n \n evi_cropped.rio.to_raster(raster_path=f'{out_folder}{outName}', driver='COG')\n \n print(f\"Processing file {j+1} of {len(hls_results_urls)}\")\n \n\nProcessing file 1 of 78\nProcessing file 2 of 78\nProcessing file 3 of 78\nProcessing file 4 of 78\nProcessing file 5 of 78\nProcessing file 6 of 78\nProcessing file 7 of 78\nProcessing file 8 of 78\nProcessing file 9 of 78\nProcessing file 10 of 78\nHLS.S30.T10TEK.2021142T185921.v2.0_EVI_cropped.tif has already been processed and is available in this directory, moving to next file.\nProcessing file 12 of 78\nProcessing file 13 of 78\nProcessing file 14 of 78\nProcessing file 15 of 78\nProcessing file 16 of 78\nProcessing file 17 of 78\nProcessing file 18 of 78\nProcessing file 19 of 78\nProcessing file 20 of 78\nProcessing file 21 of 78\nProcessing file 22 of 78\nProcessing file 23 of 78\nProcessing file 24 of 78\nProcessing file 25 of 78\nProcessing file 26 of 78\nProcessing file 27 of 78\nProcessing file 28 of 78\nProcessing file 29 of 78\nProcessing file 30 of 78\nProcessing file 31 of 78\nProcessing file 32 of 78\nProcessing file 33 of 78\nProcessing file 34 of 78\nProcessing file 35 of 78\nProcessing file 36 of 78\nProcessing file 37 of 78\nProcessing file 38 of 78\nProcessing file 39 of 78\nProcessing file 40 of 78\nProcessing file 41 of 78\nProcessing file 42 of 78\nProcessing file 43 of 78\nProcessing file 44 of 78\nProcessing file 45 of 78\nProcessing file 46 of 78\nProcessing file 47 of 78\nProcessing file 48 of 78\nProcessing file 49 of 78\nProcessing file 50 of 78\nProcessing file 51 of 78\nProcessing file 52 of 78\nProcessing file 53 of 78\nProcessing file 54 of 78\nProcessing file 55 of 78\nProcessing file 56 of 78\nProcessing file 57 of 78\nProcessing file 58 of 78\nProcessing file 59 of 78\nProcessing file 60 of 78\nProcessing file 61 of 78\nProcessing file 62 of 78\nProcessing file 63 of 78\nProcessing file 64 of 78\nProcessing file 65 of 78\nProcessing file 66 of 78\nProcessing file 67 of 78\nProcessing file 68 of 78\nProcessing file 69 of 78\nProcessing file 70 of 78\nProcessing file 71 of 78\nProcessing file 72 of 78\nProcessing file 73 of 78\nProcessing file 74 of 78\nProcessing file 75 of 78\nProcessing file 76 of 78\nProcessing file 77 of 78\nProcessing file 78 of 78\n\n\nNow there should be multiple COGs exported to your working directory, that will be used in Section 6 to stack into a time series." + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#access-s3-storage-location", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#access-s3-storage-location", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Access s3 storage location", + "text": "Access s3 storage location\nAccess s3 credentials from LP.DAAC and create a boto3 Session object using your temporary credentials. This Session is used to pass credentials and configuration to AWS so we can interact wit S3 objects from applicable buckets.\n\ns3_cred_endpoint = 'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials'\ntemp_creds_req = requests.get(s3_cred_endpoint).json()\n\nsession = boto3.Session(aws_access_key_id=temp_creds_req['accessKeyId'], \n aws_secret_access_key=temp_creds_req['secretAccessKey'],\n aws_session_token=temp_creds_req['sessionToken'],\n region_name='us-west-2')\n\nGDAL Configurations\nGDAL is a foundational piece of geospatial software that is leveraged by several popular open-source, and closed, geospatial software. The rasterio package is no exception. Rasterio leverages GDAL to, among other things, read and write raster data files, e.g., GeoTIFFs/Cloud Optimized GeoTIFFs. To read remote files, i.e., files/objects stored in the cloud, GDAL uses its Virtual File System API. In a perfect world, one would be able to point a Virtual File System (there are several) at a remote data asset and have the asset retrieved, but that is not always the case. GDAL has a host of configurations/environmental variables that adjust its behavior to, for example, make a request more performant or to pass AWS credentials to the distribution system. Below, we’ll identify the evironmental variables that will help us get our data from cloud\n\nrio_env = rio.Env(AWSSession(session),\n GDAL_DISABLE_READDIR_ON_OPEN='TRUE',\n GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/cookies.txt'),\n GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/cookies.txt'))\nrio_env.__enter__()\n\n<rasterio.env.Env at 0x7f64510812e0>\n\n\n\ns3_url = 's3://lp-prod-protected/HLSL30.020/HLS.L30.T11SQA.2021333T181532.v2.0/HLS.L30.T11SQA.2021333T181532.v2.0.B04.tif'\n# s3_url = 's3://lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B04.tif'" }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#stacking-hls-data", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#stacking-hls-data", - "title": "Observing Seasonality in Agricultural Areas", - "section": "6. Stacking HLS Data ", - "text": "6. Stacking HLS Data \nIn this section we will open multiple HLS-derived EVI COGs and stack them into an xarray data array along the time dimension. First list the files we created in the /data/ directory.\n\n6.1 Open and Stack COGs\n\nevi_files = [os.path.abspath(o) for o in os.listdir(out_folder) if o.endswith('EVI_cropped.tif')] # List EVI COGs\n\nCreate a time index as an xarray variable from the filenames.\n\ndef time_index_from_filenames(evi_files):\n '''\n Helper function to create a pandas DatetimeIndex\n '''\n return [datetime.strptime(f.split('.')[-4], '%Y%jT%H%M%S') for f in evi_files]\n\ntime = xr.Variable('time', time_index_from_filenames(evi_files))\n\nNext, the cropped HLS COG files are being read using rioxarray and a time series stack is created using xarray.\n\n#evi_ts = evi_ts.squeeze('band', drop=True)\nchunks=dict(band=1, x=512, y=512)\n\nevi_ts = xr.concat([rxr.open_rasterio(f, mask_and_scale=True, chunks=chunks).squeeze('band', drop=True) for f in evi_files], dim=time)\nevi_ts.name = 'EVI'\n\n\nevi_ts = evi_ts.sortby(evi_ts.time)\nevi_ts\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray 'EVI' (time: 78, y: 59, x: 38)>\ndask.array<getitem, shape=(78, 59, 38), dtype=float32, chunksize=(1, 59, 38), chunktype=numpy.ndarray>\nCoordinates:\n * x (x) float64 5.802e+05 5.802e+05 ... 5.812e+05 5.813e+05\n * y (y) float64 4.419e+06 4.419e+06 ... 4.417e+06 4.417e+06\n spatial_ref int32 0\n * time (time) datetime64[ns] 2021-05-02T18:59:11 ... 2021-09-29T19:...\nAttributes: (12/33)\n ACCODE: Lasrc; Lasrc\n AREA_OR_POINT: Area\n arop_ave_xshift(meters): 0, 0\n arop_ave_yshift(meters): 0, 0\n arop_ncp: 0, 0\n arop_rmse(meters): 0, 0\n ... ...\n SPATIAL_RESOLUTION: 30\n TIRS_SSM_MODEL: FINAL; FINAL\n TIRS_SSM_POSITION_STATUS: ESTIMATED; ESTIMATED\n ULX: 499980\n ULY: 4500000\n USGS_SOFTWARE: LPGS_15.4.0xarray.DataArray'EVI'time: 78y: 59x: 38dask.array<chunksize=(1, 59, 38), meta=np.ndarray>\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n683.11 kiB\n8.76 kiB\n\n\nShape\n(78, 59, 38)\n(1, 59, 38)\n\n\nDask graph\n78 chunks in 314 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\nCoordinates: (4)x(x)float645.802e+05 5.802e+05 ... 5.813e+05array([580155., 580185., 580215., 580245., 580275., 580305., 580335., 580365.,\n 580395., 580425., 580455., 580485., 580515., 580545., 580575., 580605.,\n 580635., 580665., 580695., 580725., 580755., 580785., 580815., 580845.,\n 580875., 580905., 580935., 580965., 580995., 581025., 581055., 581085.,\n 581115., 581145., 581175., 581205., 581235., 581265.])y(y)float644.419e+06 4.419e+06 ... 4.417e+06array([4418535., 4418505., 4418475., 4418445., 4418415., 4418385., 4418355.,\n 4418325., 4418295., 4418265., 4418235., 4418205., 4418175., 4418145.,\n 4418115., 4418085., 4418055., 4418025., 4417995., 4417965., 4417935.,\n 4417905., 4417875., 4417845., 4417815., 4417785., 4417755., 4417725.,\n 4417695., 4417665., 4417635., 4417605., 4417575., 4417545., 4417515.,\n 4417485., 4417455., 4417425., 4417395., 4417365., 4417335., 4417305.,\n 4417275., 4417245., 4417215., 4417185., 4417155., 4417125., 4417095.,\n 4417065., 4417035., 4417005., 4416975., 4416945., 4416915., 4416885.,\n 4416855., 4416825., 4416795.])spatial_ref()int320crs_wkt :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not specified (based on WGS 84 spheroid)projected_crs_name :UTM Zone 10, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-123.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 10, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not specified (based on WGS 84 spheroid)\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-123],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :580140.0 30.0 0.0 4418550.0 0.0 -30.0array(0)time(time)datetime64[ns]2021-05-02T18:59:11 ... 2021-09-...array(['2021-05-02T18:59:11.000000000', '2021-05-04T18:49:19.000000000',\n '2021-05-07T18:59:19.000000000', '2021-05-08T18:44:47.000000000',\n '2021-05-09T18:49:21.000000000', '2021-05-12T18:59:21.000000000',\n '2021-05-14T18:49:19.000000000', '2021-05-15T18:51:03.000000000',\n '2021-05-17T18:59:19.000000000', '2021-05-19T18:49:21.000000000',\n '2021-05-22T18:59:21.000000000', '2021-05-24T18:44:57.000000000',\n '2021-05-24T18:49:19.000000000', '2021-05-27T18:59:19.000000000',\n '2021-05-29T18:49:21.000000000', '2021-05-31T18:51:12.000000000',\n '2021-06-01T18:59:21.000000000', '2021-06-03T18:49:19.000000000',\n '2021-06-06T18:59:19.000000000', '2021-06-08T18:49:21.000000000',\n '2021-06-09T18:45:05.000000000', '2021-06-11T18:59:21.000000000',\n '2021-06-13T18:49:19.000000000', '2021-06-16T18:51:18.000000000',\n '2021-06-16T18:59:19.000000000', '2021-06-18T18:49:21.000000000',\n '2021-06-21T18:59:21.000000000', '2021-06-23T18:49:19.000000000',\n '2021-06-25T18:45:09.000000000', '2021-06-26T18:59:19.000000000',\n '2021-06-28T18:49:21.000000000', '2021-07-01T18:59:21.000000000',\n '2021-07-02T18:51:21.000000000', '2021-07-06T18:59:19.000000000',\n '2021-07-08T18:49:21.000000000', '2021-07-11T18:45:11.000000000',\n '2021-07-11T18:59:21.000000000', '2021-07-13T18:49:19.000000000',\n '2021-07-16T18:59:19.000000000', '2021-07-18T18:49:21.000000000',\n '2021-07-18T18:51:24.000000000', '2021-07-21T18:59:21.000000000',\n '2021-07-23T18:49:19.000000000', '2021-07-26T18:59:19.000000000',\n '2021-07-28T18:49:21.000000000', '2021-07-31T18:59:21.000000000',\n '2021-08-02T18:49:19.000000000', '2021-08-03T18:51:32.000000000',\n '2021-08-05T18:59:19.000000000', '2021-08-07T18:49:21.000000000',\n '2021-08-10T18:59:21.000000000', '2021-08-12T18:45:24.000000000',\n '2021-08-12T18:49:19.000000000', '2021-08-15T18:59:19.000000000',\n '2021-08-17T18:49:21.000000000', '2021-08-19T18:51:37.000000000',\n '2021-08-20T18:59:21.000000000', '2021-08-22T18:49:19.000000000',\n '2021-08-25T18:59:19.000000000', '2021-08-27T18:49:21.000000000',\n '2021-08-28T18:45:29.000000000', '2021-08-30T18:59:21.000000000',\n '2021-09-01T18:49:19.000000000', '2021-09-04T18:51:42.000000000',\n '2021-09-04T18:59:09.000000000', '2021-09-06T18:49:21.000000000',\n '2021-09-09T18:59:31.000000000', '2021-09-11T18:49:29.000000000',\n '2021-09-13T18:45:33.000000000', '2021-09-14T18:59:29.000000000',\n '2021-09-16T18:50:21.000000000', '2021-09-19T19:00:51.000000000',\n '2021-09-20T18:51:45.000000000', '2021-09-21T18:49:59.000000000',\n '2021-09-24T19:00:29.000000000', '2021-09-26T18:51:31.000000000',\n '2021-09-29T18:45:37.000000000', '2021-09-29T19:02:01.000000000'],\n dtype='datetime64[ns]')Indexes: (3)xPandasIndexPandasIndex(Index([580155.0, 580185.0, 580215.0, 580245.0, 580275.0, 580305.0, 580335.0,\n 580365.0, 580395.0, 580425.0, 580455.0, 580485.0, 580515.0, 580545.0,\n 580575.0, 580605.0, 580635.0, 580665.0, 580695.0, 580725.0, 580755.0,\n 580785.0, 580815.0, 580845.0, 580875.0, 580905.0, 580935.0, 580965.0,\n 580995.0, 581025.0, 581055.0, 581085.0, 581115.0, 581145.0, 581175.0,\n 581205.0, 581235.0, 581265.0],\n dtype='float64', name='x'))yPandasIndexPandasIndex(Index([4418535.0, 4418505.0, 4418475.0, 4418445.0, 4418415.0, 4418385.0,\n 4418355.0, 4418325.0, 4418295.0, 4418265.0, 4418235.0, 4418205.0,\n 4418175.0, 4418145.0, 4418115.0, 4418085.0, 4418055.0, 4418025.0,\n 4417995.0, 4417965.0, 4417935.0, 4417905.0, 4417875.0, 4417845.0,\n 4417815.0, 4417785.0, 4417755.0, 4417725.0, 4417695.0, 4417665.0,\n 4417635.0, 4417605.0, 4417575.0, 4417545.0, 4417515.0, 4417485.0,\n 4417455.0, 4417425.0, 4417395.0, 4417365.0, 4417335.0, 4417305.0,\n 4417275.0, 4417245.0, 4417215.0, 4417185.0, 4417155.0, 4417125.0,\n 4417095.0, 4417065.0, 4417035.0, 4417005.0, 4416975.0, 4416945.0,\n 4416915.0, 4416885.0, 4416855.0, 4416825.0, 4416795.0],\n dtype='float64', name='y'))timePandasIndexPandasIndex(DatetimeIndex(['2021-05-02 18:59:11', '2021-05-04 18:49:19',\n '2021-05-07 18:59:19', '2021-05-08 18:44:47',\n '2021-05-09 18:49:21', '2021-05-12 18:59:21',\n '2021-05-14 18:49:19', '2021-05-15 18:51:03',\n '2021-05-17 18:59:19', '2021-05-19 18:49:21',\n '2021-05-22 18:59:21', '2021-05-24 18:44:57',\n '2021-05-24 18:49:19', '2021-05-27 18:59:19',\n '2021-05-29 18:49:21', '2021-05-31 18:51:12',\n '2021-06-01 18:59:21', '2021-06-03 18:49:19',\n '2021-06-06 18:59:19', '2021-06-08 18:49:21',\n '2021-06-09 18:45:05', '2021-06-11 18:59:21',\n '2021-06-13 18:49:19', '2021-06-16 18:51:18',\n '2021-06-16 18:59:19', '2021-06-18 18:49:21',\n '2021-06-21 18:59:21', '2021-06-23 18:49:19',\n '2021-06-25 18:45:09', '2021-06-26 18:59:19',\n '2021-06-28 18:49:21', '2021-07-01 18:59:21',\n '2021-07-02 18:51:21', '2021-07-06 18:59:19',\n '2021-07-08 18:49:21', '2021-07-11 18:45:11',\n '2021-07-11 18:59:21', '2021-07-13 18:49:19',\n '2021-07-16 18:59:19', '2021-07-18 18:49:21',\n '2021-07-18 18:51:24', '2021-07-21 18:59:21',\n '2021-07-23 18:49:19', '2021-07-26 18:59:19',\n '2021-07-28 18:49:21', '2021-07-31 18:59:21',\n '2021-08-02 18:49:19', '2021-08-03 18:51:32',\n '2021-08-05 18:59:19', '2021-08-07 18:49:21',\n '2021-08-10 18:59:21', '2021-08-12 18:45:24',\n '2021-08-12 18:49:19', '2021-08-15 18:59:19',\n '2021-08-17 18:49:21', '2021-08-19 18:51:37',\n '2021-08-20 18:59:21', '2021-08-22 18:49:19',\n '2021-08-25 18:59:19', '2021-08-27 18:49:21',\n '2021-08-28 18:45:29', '2021-08-30 18:59:21',\n '2021-09-01 18:49:19', '2021-09-04 18:51:42',\n '2021-09-04 18:59:09', '2021-09-06 18:49:21',\n '2021-09-09 18:59:31', '2021-09-11 18:49:29',\n '2021-09-13 18:45:33', '2021-09-14 18:59:29',\n '2021-09-16 18:50:21', '2021-09-19 19:00:51',\n '2021-09-20 18:51:45', '2021-09-21 18:49:59',\n '2021-09-24 19:00:29', '2021-09-26 18:51:31',\n '2021-09-29 18:45:37', '2021-09-29 19:02:01'],\n dtype='datetime64[ns]', name='time', freq=None))Attributes: (33)ACCODE :Lasrc; LasrcAREA_OR_POINT :Areaarop_ave_xshift(meters) :0, 0arop_ave_yshift(meters) :0, 0arop_ncp :0, 0arop_rmse(meters) :0, 0arop_s2_refimg :NONEcloud_coverage :0HLS_PROCESSING_TIME :2021-10-14T22:05:21ZHORIZONTAL_CS_NAME :UTM, WGS84, UTM ZONE 10; UTM, WGS84, UTM ZONE 10L1_PROCESSING_TIME :2021-05-18T02:30:51Z; 2021-05-18T02:30:41ZLANDSAT_PRODUCT_ID :LC08_L1TP_044032_20210508_20210518_02_T1; LC08_L1TP_044033_20210508_20210518_02_T1LANDSAT_SCENE_ID :LC80440322021128LGN00; LC80440332021128LGN00long_name :EVIMEAN_SUN_AZIMUTH_ANGLE :137.102321035166MEAN_SUN_ZENITH_ANGLE :28.615823781336MEAN_VIEW_AZIMUTH_ANGLE :105.923057001037MEAN_VIEW_ZENITH_ANGLE :5.72350593998171NBAR_SOLAR_ZENITH :27.1549084508815NCOLS :3660NROWS :3660OVR_RESAMPLING_ALG :NEARESTPROCESSING_LEVEL :L1TP; L1TPSENSING_TIME :2021-05-08T18:44:47.4233880Z; 2021-05-08T18:45:11.3101920ZSENSOR :OLI_TIRS; OLI_TIRSSENTINEL2_TILEID :10TEKspatial_coverage :55SPATIAL_RESOLUTION :30TIRS_SSM_MODEL :FINAL; FINALTIRS_SSM_POSITION_STATUS :ESTIMATED; ESTIMATEDULX :499980ULY :4500000USGS_SOFTWARE :LPGS_15.4.0\n\n\n\n\n6.2 Visualize Stacked Time Series\nBelow, use the hvPlot and holoviews packages to create an interactive time series plot of the HLS derived EVI data.Basemap layer is also added to provide better context of the areas surrounding our region of interest.\n\n# This cell generates a warning\nimport warnings\nwarnings.filterwarnings('ignore')\n\n\n# set the x, y, and z (groupby) dimensions, add a colormap/bar and other parameters.\ntitle = 'HLS-derived EVI over agricultural fields in northern California'\nevi_ts.hvplot.image(x='x', y='y', groupby = 'time', frame_height=500, frame_width= 500, cmap='YlGn', geo=True, tiles = 'EsriImagery',)\n\n\n\n\n\n \n\n\n\n\nLooking at the time series above, the farm fields are pretty stable in terms of EVI during our temporal range. The slow decrease in EVI as we move toward Fall season could show these fields are having some sort of trees rather than crops. I encourage you to expand your temporal range to learn more about the EVI annual and seasonal changes.\nSince the data is in an xarray we can intuitively slice or reduce the dataset. Let’s select a single time slice from the EVI variable.\nYou can use slicing to plot data only for a specific observation, for example.\n\ntitle = 'HLS-derived EVI over agricultural fields in northern California'\n# evi_cropped.hvplot.image(aspect='equal', cmap='YlGn', frame_width=300).opts(title=f'HLS-derived EVI, {evi_cropped.SENSING_TIME}', clabel='EVI')\n\nevi_ts.isel(time=1).hvplot.image(x='x', y='y', cmap='YlGn', geo=True, tiles = 'EsriImagery', frame_height=500, frame_width= 500).opts(title=f'{title}, {evi_ts.isel(time=4).SENSING_TIME}')\n\n\n\n\n\n \n\n\n\n\nNow, plot the time series as boxplots showing the distribution of EVI values for our farm field.\n\nevi_ts.hvplot.box('EVI', by=['time'], rot=90, box_fill_color='lightblue', width=800, height=600).opts(ylim=(0,1.0))\n\n\n\n\n\n \n\n\n\n\nThe statistics shows a relatively stable green status in these fields during mid May to the end of September 2021." + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#read-cloud-optimized-geotiff-into-rioxarray", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#read-cloud-optimized-geotiff-into-rioxarray", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Read Cloud-Optimized GeoTIFF into rioxarray", + "text": "Read Cloud-Optimized GeoTIFF into rioxarray\n\nda = rioxarray.open_rasterio(s3_url)\nda\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (band: 1, y: 3660, x: 3660)>\n[13395600 values with dtype=int16]\nCoordinates:\n * band (band) int64 1\n * x (x) float64 7e+05 7e+05 7e+05 ... 8.097e+05 8.097e+05 8.097e+05\n * y (y) float64 4.1e+06 4.1e+06 4.1e+06 ... 3.99e+06 3.99e+06\n spatial_ref int64 0\nAttributes:\n _FillValue: -9999.0\n scale_factor: 0.0001\n add_offset: 0.0\n long_name: Redxarray.DataArrayband: 1y: 3660x: 3660...[13395600 values with dtype=int16]Coordinates: (4)band(band)int641array([1])x(x)float647e+05 7e+05 ... 8.097e+05 8.097e+05array([699975., 700005., 700035., ..., 809685., 809715., 809745.])y(y)float644.1e+06 4.1e+06 ... 3.99e+06array([4100025., 4099995., 4099965., ..., 3990315., 3990285., 3990255.])spatial_ref()int640crs_wkt :PROJCS[\"UTM Zone 11, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not_specified_based_on_WGS_84_spheroid\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-117],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not_specified_based_on_WGS_84_spheroidprojected_crs_name :UTM Zone 11, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-117.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 11, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not_specified_based_on_WGS_84_spheroid\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-117],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :699960.0 30.0 0.0 4100040.0 0.0 -30.0array(0)Attributes: (4)_FillValue :-9999.0scale_factor :0.0001add_offset :0.0long_name :Red\n\n\nWhen GeoTIFFS/Cloud Optimized GeoTIFFS are read in, a band coordinate variable is automatically created (see the print out above). In this exercise we will not use that coordinate variable, so we will remove it using the squeeze() function to avoid confusion.\n\nda_red = da.squeeze('band', drop=True)\nda_red\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.DataArray (y: 3660, x: 3660)>\n[13395600 values with dtype=int16]\nCoordinates:\n * x (x) float64 7e+05 7e+05 7e+05 ... 8.097e+05 8.097e+05 8.097e+05\n * y (y) float64 4.1e+06 4.1e+06 4.1e+06 ... 3.99e+06 3.99e+06\n spatial_ref int64 0\nAttributes:\n _FillValue: -9999.0\n scale_factor: 0.0001\n add_offset: 0.0\n long_name: Redxarray.DataArrayy: 3660x: 3660...[13395600 values with dtype=int16]Coordinates: (3)x(x)float647e+05 7e+05 ... 8.097e+05 8.097e+05array([699975., 700005., 700035., ..., 809685., 809715., 809745.])y(y)float644.1e+06 4.1e+06 ... 3.99e+06array([4100025., 4099995., 4099965., ..., 3990315., 3990285., 3990255.])spatial_ref()int640crs_wkt :PROJCS[\"UTM Zone 11, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not_specified_based_on_WGS_84_spheroid\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-117],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]semi_major_axis :6378137.0semi_minor_axis :6356752.314245179inverse_flattening :298.257223563reference_ellipsoid_name :WGS 84longitude_of_prime_meridian :0.0prime_meridian_name :Greenwichgeographic_crs_name :Unknown datum based upon the WGS 84 ellipsoidhorizontal_datum_name :Not_specified_based_on_WGS_84_spheroidprojected_crs_name :UTM Zone 11, Northern Hemispheregrid_mapping_name :transverse_mercatorlatitude_of_projection_origin :0.0longitude_of_central_meridian :-117.0false_easting :500000.0false_northing :0.0scale_factor_at_central_meridian :0.9996spatial_ref :PROJCS[\"UTM Zone 11, Northern Hemisphere\",GEOGCS[\"Unknown datum based upon the WGS 84 ellipsoid\",DATUM[\"Not_specified_based_on_WGS_84_spheroid\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-117],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]GeoTransform :699960.0 30.0 0.0 4100040.0 0.0 -30.0array(0)Attributes: (4)_FillValue :-9999.0scale_factor :0.0001add_offset :0.0long_name :Red" }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#export-statistics", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#export-statistics", - "title": "Observing Seasonality in Agricultural Areas", - "section": "7. Export Statistics", - "text": "7. Export Statistics\nNext, calculate statistics for each observation and export to CSV.\n\n# xarray allows you to easily calculate a number of statistics\nevi_min = evi_ts.min(('y', 'x'))\nevi_max = evi_ts.max(('y', 'x'))\nevi_mean = evi_ts.mean(('y', 'x'))\nevi_sd = evi_ts.std(('y', 'x'))\nevi_count = evi_ts.count(('y', 'x'))\nevi_median = evi_ts.median(('y', 'x'))\n\nWe now have the mean and standard deviation for each time slice as well as the maximum and minimum values. Let’s do some plotting! We will use the hvPlot package to create simple but interactive charts/plots. Hover your curser over the visualization to see the data values.\n\nevi_mean.hvplot.line()\n\n\n\n\n\n \n\n\n\n\n\n# Ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Combine line plots for different statistics\nstats = (evi_mean.hvplot.line(height=350, width=450, line_width=1.5, color='red', grid=True, padding=0.05).opts(title='Mean')+ \n evi_sd.hvplot.line(height=350, width=450, line_width=1.5, color='red', grid=True, padding=0.05).opts(title='Standard Deviation')\n + evi_max.hvplot.line(height=350, width=450, line_width=1.5, color='red', grid=True, padding=0.05).opts(title='Max') + \n evi_min.hvplot.line(height=350, width=450, line_width=1.5, color='red', grid=True, padding=0.05).opts(title='Min')).cols(2)\nstats\n\n\n\n\n\n \n\n\n\n\nRemember that these graphs are also interactive–hover over the line to see the value for a given date.\nFinally, create a pandas dataframe with the statistics, and export to a CSV file.\n\n# Create pandas dataframe from dictionary\ndf = pd.DataFrame({'Min EVI': evi_min, 'Max EVI': evi_max, \n 'Mean EVI': evi_mean, 'Standard Deviation EVI': evi_sd, \n 'Median EVI': evi_median, 'Count': evi_count})\n\n\ndf.index = evi_ts.time.data # Set the observation date as the index\ndf.to_csv(f'{out_folder}HLS-Derived_EVI_Stats.csv', index=True) # Unhash to export to CSV\n\n\nNow remove the output files to clean up the workspace.\n\nimport shutil\nshutil.rmtree(os.path.abspath(out_folder)) # Execute once ready to remove output files\n\nSuccess! You have now not only learned how to get started with HLS V2.0 data, but have also learned how to navigate cloud-native data using STAC, how to access subsets of COGs, and how to write COGs for your own outputs. Using this jupyter notebook as a workflow, you should now be able to switch to your specific region of interest and re-run the notebook. Good Luck!" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#plot-using-hvplot", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#plot-using-hvplot", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Plot using hvplot", + "text": "Plot using hvplot\n\nda_red.hvplot.image(x='x', y='y', cmap='gray', aspect='equal')\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\nrio_env.__exit__()" }, { - "objectID": "tutorials/Observing_Seasonal_Ag_Changes.html#contact-info", - "href": "tutorials/Observing_Seasonal_Ag_Changes.html#contact-info", - "title": "Observing Seasonality in Agricultural Areas", - "section": "Contact Info", - "text": "Contact Info\nEmail: LPDAAC@usgs.gov\nVoice: +1-866-573-3222\nOrganization: Land Processes Distributed Active Archive Center (LP DAAC)¹\nWebsite: https://lpdaac.usgs.gov/\nDate last modified: 09-08-2023\n¹Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I." + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#part-3-working-with-zarr-formatted-data-using-nasas-harmony-cloud-transformation-service", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#part-3-working-with-zarr-formatted-data-using-nasas-harmony-cloud-transformation-service", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Part 3: Working with Zarr-formatted data using NASA’s Harmony cloud transformation service", + "text": "Part 3: Working with Zarr-formatted data using NASA’s Harmony cloud transformation service\nWe have already explored direct access to the NASA EOSDIS archive in the cloud via the Amazon Simple Storage Service (S3). In addition to directly accessing the files archived and distributed by each of the NASA DAACs, many datasets also support services that allow us to customize the data via subsetting, reformatting, reprojection, and other transformations.\nThis example demonstrates “analysis in place” using customized ECCO Level 4 monthly sea surface height data, in this case reformatted to Zarr, from a new ecosystem of services operating within the NASA Earthdata Cloud: NASA Harmony:\n\nConsistent access patterns to EOSDIS holdings make cross-data center data access easier\nData reduction services allow us to request only the data we want, in the format and projection we want\nAnalysis Ready Data and cloud access will help reduce time-to-science\nCommunity Development helps reduce the barriers for re-use of code and sharing of domain knowledge" }, { - "objectID": "tutorials/04_On-Prem_Cloud.html#summary", - "href": "tutorials/04_On-Prem_Cloud.html#summary", - "title": "Pairing Cloud and non-Cloud Data", - "section": "Summary", - "text": "Summary\n\nThis tutorial will combine several workflow steps and components from the previous days, demonstrating the process of using the geolocation of data available outside of the Earthdata Cloud to then access coincident variables of cloud-accessible data. This may be a common use case as NASA Earthdata continues to migrate to the cloud, producing a “hybrid” data archive across Amazon Web Services (AWS) and original on-premise data storage systems. Additionally, you may also want to combine field measurements with remote sensing data available on the Earthdata Cloud.\nThis specific example explores the pairing of the ICESat-2 ATL07 Sea Ice Height data product, currently (as of November 2021) available publicly via direct download at the NSIDC DAAC, along with Sea Surface Temperature (SST) from the GHRSST MODIS L2 dataset (MODIS_A-JPL-L2P-v2019.0) available from PO.DAAC on the Earthdata Cloud.\nThe use case we’re looking at today centers over an area north of Greenland for a single day in June, where a melt pond was observed using the NASA OpenAltimetry application. Melt ponds are an important feature of Arctic sea ice dynamics, leading to an decrease in sea ice albedo and other changes in heat balance. Many NASA Earthdata datasets produce variables including sea ice albedo, sea surface temperature, air temperature, and sea ice height, which can be used to better understand these dynamics." + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#import-packages-1", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#import-packages-1", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Import packages", + "text": "Import packages\n\nfrom harmony import BBox, Client, Collection, Request, LinkType\nfrom harmony.config import Environment\nfrom pprint import pprint\nimport datetime as dt\nimport s3fs\nfrom pqdm.threads import pqdm\nimport xarray as xr" }, { - "objectID": "tutorials/04_On-Prem_Cloud.html#requirements", - "href": "tutorials/04_On-Prem_Cloud.html#requirements", - "title": "Pairing Cloud and non-Cloud Data", - "section": "Requirements", - "text": "Requirements\n\n\nAWS instance running in us-west 2\nEarthdata Login\n.netrc file" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#using-harmony-py-to-customize-data", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#using-harmony-py-to-customize-data", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Using Harmony-Py to customize data", + "text": "Using Harmony-Py to customize data\nHarmony-Py provides a pip installable Python alternative to directly using Harmony’s RESTful API to make it easier to request data and service options, especially when interacting within a Python Jupyter Notebook environment.\n\nCreate Harmony Client object\nFirst, we need to create a Harmony Client, which is what we will interact with to submit and inspect a data request to Harmony, as well as to retrieve results.\n\nharmony_client = Client()" }, { - "objectID": "tutorials/04_On-Prem_Cloud.html#learning-objectives", - "href": "tutorials/04_On-Prem_Cloud.html#learning-objectives", - "title": "Pairing Cloud and non-Cloud Data", - "section": "Learning Objectives", - "text": "Learning Objectives\n\n\nSearch for data programmatically using the Common Metadata Repository (CMR), determining granule (file) coverage across two datasets over an area of interest.\nDownload data from an on-premise storage system to our cloud environment.\nRead in 1-dimensional trajectory data (ICESat-2 ATL07) into xarray and perform attribute conversions.\nSelect and read in sea surface temperature (SST) data (MODIS_A-JPL-L2P-v2019.0) from the Earthdata Cloud into xarray.\nExtract, resample, and plot coincident SST data based on ICESat-2 geolocation." + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#create-harmony-request", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#create-harmony-request", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Create Harmony Request", + "text": "Create Harmony Request\nSpecify a temporal range over 2015, and Zarr as an output format.\nZarr is an open source library for storing N-dimensional array data. It supports multidimensional arrays with attributes and dimensions similar to NetCDF4, and it can be read by XArray. Zarr is often used for data held in cloud object storage (like Amazon S3), because it is better optimized for these situations than NetCDF4.\n\nshort_name = 'ECCO_L4_SSH_05DEG_MONTHLY_V4R4'\n\nrequest = Request(\n collection=Collection(id=short_name),\n temporal={\n 'start': dt.datetime(2015, 1, 2),\n 'stop': dt.datetime(2015, 12, 31),\n },\n format='application/x-zarr'\n)\n\njob_id = harmony_client.submit(request)" }, { - "objectID": "tutorials/04_On-Prem_Cloud.html#import-packages", - "href": "tutorials/04_On-Prem_Cloud.html#import-packages", - "title": "Pairing Cloud and non-Cloud Data", - "section": "Import packages", - "text": "Import packages\n\nimport os\nfrom pathlib import Path\nfrom pprint import pprint\n\n# Access via download\nimport requests\n\n# Access AWS S3\nimport s3fs\n\n# Read and work with datasets\nimport xarray as xr\nimport numpy as np\nimport h5py\n\n# For plotting\nimport matplotlib.pyplot as plt\nimport cartopy.crs as ccrs\nfrom shapely.geometry import box\n\n# For resampling\nimport pyresample" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#check-request-status-and-view-output-urls", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#check-request-status-and-view-output-urls", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Check request status and view output URLs", + "text": "Check request status and view output URLs\nHarmony data outputs can be accessed within the cloud using the s3 URLs and AWS credentials provided in the Harmony job response:\n\nharmony_client.wait_for_processing(job_id, show_progress=True)\n\nresults = harmony_client.result_urls(job_id, link_type=LinkType.s3)\ns3_urls = list(results)\ns3_urls\n\n [ Processing: 83% ] |########################################## | [/]\n\n\n\nAWS credential retrieval\nUsing aws_credentials you can retrieve the credentials needed to access the Harmony s3 staging bucket and its contents.\n\ncreds = harmony_client.aws_credentials()" }, { - "objectID": "tutorials/04_On-Prem_Cloud.html#specify-data-time-range-and-area-of-interest", - "href": "tutorials/04_On-Prem_Cloud.html#specify-data-time-range-and-area-of-interest", - "title": "Pairing Cloud and non-Cloud Data", - "section": "Specify data, time range, and area of interest", - "text": "Specify data, time range, and area of interest\nWe are going to focus on getting data for an area north of Greenland for a single day in June.\nThese bounding_box and temporal variables will be used for data search, subset, and access below.\nThe same search and access steps for both datasets can be performed via Earthdata Search using the same spatial and temporal filtering options. See the Earthdata Search tutorial for more information on how to use Earthdata Search to discover and access data from the Earthdata Cloud.\n\n# Bounding Box spatial parameter in decimal degree 'W,S,E,N' format.\nbounding_box = '-62.8,81.7,-56.4,83'\n\n# Each date in yyyy-MM-ddTHH:mm:ssZ format; date range in start,end format\ntemporal = '2019-06-22T00:00:00Z,2019-06-22T23:59:59Z'\n\nSee the Data Discovery with CMR tutorial for more details on how to navigate the NASA Common Metadata Repository (CMR) Application Programming Interface, or API. For some background, the CMR catalogs all data for NASA’s Earth Observing System Data and Information System (EOSDIS). The CMR API allows applications (software, services, etc) to send information to each other. A helpful analogy is a waiter in a restaurant. The waiter takes your drink or food order that you select from the menu, often translated into short-hand, to the bar or kitchen, and then returns (hopefully) with what you ordered when it is ready.\nThe CMR API accepts search terms such as collection name, keywords, datetime range, and location, queries the CMR database and returns the results.\nFor this tutorial, we have already identified the unique identifier, or concept_id for each dataset:\n\nmodis_concept_id = 'C1940473819-POCLOUD'\nicesat2_concept_id = 'C2003771980-NSIDC_ECS'\n\nThis Earthdata Search Project also provides the same data access links that we will identify in the following steps for each dataset (note that you will need an Earthdata Login account to access this project)." + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#open-staged-files-with-s3fs-and-xarray", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#open-staged-files-with-s3fs-and-xarray", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Open staged files with s3fs and xarray", + "text": "Open staged files with s3fs and xarray\nAccess AWS credentials for the Harmony bucket, and use the AWS s3fs package to create a file system that can then be read by xarray. Below we create session by passing in the temporary credentials we recieved from our temporary credentials endpoint.\n\ncreds = harmony_client.aws_credentials()\n\ns3_fs = s3fs.S3FileSystem(\n key=creds['aws_access_key_id'],\n secret=creds['aws_secret_access_key'],\n token=creds['aws_session_token'],\n client_kwargs={'region_name':'us-west-2'},\n)\n\nOpen the Zarr stores using the s3fs package, then load them all at once into a concatenated xarray dataset:\n\nstores = [s3fs.S3Map(root=url, s3=s3_fs, check=False) for url in s3_urls]\ndef open_zarr_xarray(store):\n return xr.open_zarr(store=store, consolidated=True)\n\ndatasets = pqdm(stores, open_zarr_xarray, n_jobs=12)\n\nds = xr.concat(datasets, 'time', coords='minimal', )\nds = xr.decode_cf(ds, mask_and_scale=True, decode_coords=True)\nds\n\n\nssh_da = ds.SSH\n\nssh_da.to_masked_array(copy=False)\n\nssh_da" }, { - "objectID": "tutorials/04_On-Prem_Cloud.html#search-and-download-icesat-2-atl07-files", - "href": "tutorials/04_On-Prem_Cloud.html#search-and-download-icesat-2-atl07-files", - "title": "Pairing Cloud and non-Cloud Data", - "section": "Search and download ICESat-2 ATL07 files", - "text": "Search and download ICESat-2 ATL07 files\nPerform a granule search over our time and area of interest. How many granules are returned?\n\ngranule_url = 'https://cmr.earthdata.nasa.gov/search/granules'\n\n\nresponse = requests.get(granule_url,\n params={\n 'concept_id': icesat2_concept_id,\n 'temporal': temporal,\n 'bounding_box': bounding_box,\n 'page_size': 200,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\nprint(response.headers['CMR-Hits'])\n\n2\n\n\nPrint the file names, size, and links:\n\ngranules = response.json()['feed']['entry']\nfor granule in granules:\n print(f'{granule[\"producer_granule_id\"]} {granule[\"granule_size\"]} {granule[\"links\"][0][\"href\"]}')\n\nATL07-01_20190622055317_12980301_004_01.h5 237.0905504227 https://n5eil01u.ecs.nsidc.org/DP9/ATLAS/ATL07.004/2019.06.22/ATL07-01_20190622055317_12980301_004_01.h5\nATL07-01_20190622200154_13070301_004_01.h5 230.9151573181 https://n5eil01u.ecs.nsidc.org/DP9/ATLAS/ATL07.004/2019.06.22/ATL07-01_20190622200154_13070301_004_01.h5\n\n\n\nDownload ATL07 files\nAlthough several services are supported for ICESat-2 data, we are demonstrating direct access through the “on-prem” file system at NSIDC for simplicity.\nSome of these services include: - icepyx - From the icepyx documentation: “icepyx is both a software library and a community composed of ICESat-2 data users, developers, and the scientific community. We are working together to develop a shared library of resources - including existing resources, new code, tutorials, and use-cases/examples - that simplify the process of querying, obtaining, analyzing, and manipulating ICESat-2 datasets to enable scientific discovery.” - NSIDC DAAC Data Access and Service API - The API provided by the NSIDC DAAC allows you to access data programmatically using specific temporal and spatial filters. The same subsetting, reformatting, and reprojection services available on select data sets through NASA Earthdata Search can also be applied using this API. - IceFlow - The IceFlow python library simplifies accessing and combining data from several of NASA’s cryospheric altimetry missions, including ICESat/GLAS, Operation IceBridge, and ICESat-2. In particular, IceFlow harmonizes the various file formats and georeferencing parameters across several of the missions’ data sets, allowing you to analyze data across the multi-decadal time series.\nWe’ve found 2 granules. We’ll download the first one and write it to a file with the same name as the producer_granule_id.\nWe need the url for the granule as well. This is href links we printed out above.\n\nicesat_id = granules[0]['producer_granule_id']\nicesat_url = granules[0]['links'][0]['href']\n\nTo retrieve the granule data, we use the requests.get() method, which will utilize the .netrc file on the backend to authenticate the request against Earthdata Login.\n\nr = requests.get(icesat_url)\n\nThe response returned by requests has the same structure as all the other responses: a header and contents. The header information has information about the response, including the size of the data we downloaded in bytes.\n\nfor k, v in r.headers.items():\n print(f'{k}: {v}')\n\nDate: Sun, 12 Dec 2021 01:52:31 GMT\nServer: Apache\nVary: User-Agent\nContent-Disposition: attachment\nContent-Length: 248607461\nKeep-Alive: timeout=15, max=100\nConnection: Keep-Alive\n\n\nThe contents needs to be saved to a file. To keep the directory clean, we will create a downloads directory to store the file. We can use a shell command to do this or use the makedirs method from the os package.\n\nos.makedirs(\"downloads\", exist_ok=True)\n\nYou should see a downloads directory in the file browser.\nTo write the data to a file, we use open to open a file. We need to specify that the file is open for writing by using the write-mode w. We also need to specify that we want to write bytes by setting the binary-mode b. This is important because the response contents are bytes. The default mode for open is text-mode. So make sure you use b.\nWe’ll use the with statement context-manager to open the file, write the contents of the response, and then close the file. Once the data in r.content is written sucessfully to the file, or if there is an error, the file is closed by the context-manager.\nWe also need to prepend the downloads path to the filename. We do this using Path from the pathlib package in the standard library.\n\noutfile = Path('downloads', icesat_id)\n\n\nif not outfile.exists():\n with open(outfile, 'wb') as f:\n f.write(r.content)\n\nATL07-01_20190622055317_12980301_004_01.h5 is an HDF5 file. xarray can open this but you need to tell it which group to read the data from. In this case we read the sea ice segment height data for ground-track 1 left-beam. You can explore the variable hierarchy in Earthdata Search, by selecting the Customize option under Download Data.\nThis code block performs the following operations: - Extracts the height_segment_height variable from the heights group, along with the dimension variables contained in the higher level sea_ice_segments group, - Convert attributes from bytestrings to strings, - Drops the HDF attribute DIMENSION_LIST, - Sets _FillValue to NaN\n\nvariable_names = [\n '/gt1l/sea_ice_segments/latitude',\n '/gt1l/sea_ice_segments/longitude',\n '/gt1l/sea_ice_segments/delta_time',\n '/gt1l/sea_ice_segments/heights/height_segment_height'\n ]\nwith h5py.File(outfile, 'r') as h5:\n data_vars = {}\n for varname in variable_names:\n var = h5[varname]\n name = varname.split('/')[-1]\n # Convert attributes\n attrs = {}\n for k, v in var.attrs.items():\n if k != 'DIMENSION_LIST':\n if isinstance(v, bytes):\n attrs[k] = v.decode('utf-8')\n else:\n attrs[k] = v\n data = var[:]\n if '_FillValue' in attrs:\n data = np.where(data < attrs['_FillValue'], data, np.nan)\n data_vars[name] = (['segment'], data, attrs)\n is2_ds = xr.Dataset(data_vars)\n \nis2_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (segment: 235584)\nDimensions without coordinates: segment\nData variables:\n latitude (segment) float64 82.38 82.38 82.38 ... 72.61 72.61\n longitude (segment) float64 -55.11 -55.11 ... 145.1 145.1\n delta_time (segment) float64 4.642e+07 4.642e+07 ... 4.642e+07\n height_segment_height (segment) float32 nan nan nan ... -0.4335 -0.4463xarray.DatasetDimensions:segment: 235584Coordinates: (0)Data variables: (4)latitude(segment)float6482.38 82.38 82.38 ... 72.61 72.61contentType :referenceInformationcoordinates :delta_time longitudedescription :Latitude, WGS84, North=+, Lat of segment centerlong_name :Latitudesource :ATBD, section 4.4standard_name :latitudeunits :degrees_northvalid_max :90.0valid_min :-90.0array([82.38431982, 82.38431982, 82.38431982, ..., 72.60984638,\n 72.60977493, 72.60970985])longitude(segment)float64-55.11 -55.11 ... 145.1 145.1contentType :referenceInformationcoordinates :delta_time latitudedescription :Longitude, WGS84, East=+,Lon of segment centerlong_name :Longitudesource :ATBD, section 4.4standard_name :longitudeunits :degrees_eastvalid_max :180.0valid_min :-180.0array([-55.10896068, -55.10896068, -55.10896068, ..., 145.05396164,\n 145.05392851, 145.05389832])delta_time(segment)float644.642e+07 4.642e+07 ... 4.642e+07CLASS :DIMENSION_SCALENAME :gt1l/sea_ice_segments/delta_timeREFERENCE_LIST :[(<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)]contentType :physicalMeasurementcoordinates :latitude longitudedescription :Number of GPS seconds since the ATLAS SDP epoch. The ATLAS Standard Data Products (SDP) epoch offset is defined within /ancillary_data/atlas_sdp_gps_epoch as the number of GPS seconds between the GPS epoch (1980-01-06T00:00:00.000000Z UTC) and the ATLAS SDP epoch. By adding the offset contained within atlas_sdp_gps_epoch to delta time parameters, the time in gps_seconds relative to the GPS epoch can be computed.long_name :Elapsed GPS secondssource :telemetrystandard_name :timeunits :seconds since 2018-01-01array([46419293.64266939, 46419293.64266939, 46419293.64266939, ...,\n 46419681.87646231, 46419681.87759533, 46419681.87862704])height_segment_height(segment)float32nan nan nan ... -0.4335 -0.4463_FillValue :3.4028235e+38contentType :referenceInformationcoordinates :../delta_time ../latitude ../longitudedescription :Mean height from along-track segment fit detremined by the sea ice algorithm. The sea ice height is relative to the tide-free MSS.long_name :height of segment surfacesource :ATBD, section 4.2.2.4units :metersarray([ nan, nan, nan, ..., -0.46550068,\n -0.43347716, -0.4462675 ], dtype=float32)Attributes: (0)\n\n\n\nis2_ds.height_segment_height.plot() ;" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#plot-the-sea-surface-height-time-series-using-hvplot", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#plot-the-sea-surface-height-time-series-using-hvplot", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Plot the Sea Surface Height time series using hvplot", + "text": "Plot the Sea Surface Height time series using hvplot\nNow we can start looking at aggregations across the time dimension. In this case, plot the standard deviation of the temperature at each point to get a visual sense of how much temperatures fluctuate over the course of the month.\n\nssh_da = ds.SSH\n\nstdev_ssh = ssh_da.std('time')\nstdev_ssh.name = 'stdev of analysed_sst [Kelvin]'\nstdev_ssh.plot();\n\nssh_da.hvplot.image(x='longitude', y='latitude', cmap='Spectral_r', aspect='equal').opts(clim=(ssh_da.attrs['valid_min'],ssh_da.attrs['valid_max']))" }, { - "objectID": "tutorials/04_On-Prem_Cloud.html#determine-the-ghrsst-modis-l2-granules-returned-from-our-time-and-area-of-interest", - "href": "tutorials/04_On-Prem_Cloud.html#determine-the-ghrsst-modis-l2-granules-returned-from-our-time-and-area-of-interest", - "title": "Pairing Cloud and non-Cloud Data", - "section": "Determine the GHRSST MODIS L2 granules returned from our time and area of interest", - "text": "Determine the GHRSST MODIS L2 granules returned from our time and area of interest\n\nresponse = requests.get(granule_url, \n params={\n 'concept_id': modis_concept_id,\n 'temporal': temporal,\n 'bounding_box': bounding_box,\n 'page_size': 200,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\nprint(response.headers['CMR-Hits'])\n\n14\n\n\n\ngranules = response.json()['feed']['entry']\nfor granule in granules:\n print(f'{granule[\"title\"]} {granule[\"granule_size\"]} {granule[\"links\"][0][\"href\"]}')\n\n20190622000501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 20.71552562713623 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622000501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622014501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 9.34600830078125E-5 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622014501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622032501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 21.307741165161133 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622032501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622050001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 9.34600830078125E-5 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622050001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622050501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 20.065649032592773 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622050501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622064001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 9.34600830078125E-5 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622064001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622064501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0 18.602201461791992 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622064501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\n20190622064501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 18.665077209472656 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622064501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622082001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 19.782299995422363 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622082001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622100001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 19.13440227508545 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622100001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622113501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 20.3239164352417 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622113501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622114001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 9.34600830078125E-5 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622114001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622163001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 19.257243156433105 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622163001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622181001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 19.93498420715332 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622181001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc" + "objectID": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#further-resources", + "href": "how-tos/Earthdata_Cloud__Open-Science-Tutorial.html#further-resources", + "title": "AGU 2021 Open Science in Action Tutorials:", + "section": "Further Resources", + "text": "Further Resources\n\nReference Hackathon/workshop tutorials that go into more detail!\nEarthdata Cloud Cookbook\nEarthdata Cloud Primer\n\nGetting started with Amazon Web Services outside of the Workshop to access and work with data with a cloud environment." }, { - "objectID": "tutorials/04_On-Prem_Cloud.html#load-data-into-xarray-via-s3-direct-access", - "href": "tutorials/04_On-Prem_Cloud.html#load-data-into-xarray-via-s3-direct-access", - "title": "Pairing Cloud and non-Cloud Data", - "section": "Load data into xarray via S3 direct access", - "text": "Load data into xarray via S3 direct access\nOur CMR granule search returned 14 files for our time and area of interest. However, not all granules will be suitable for analysis.\nI’ve identified the image with granule id G1956158784-POCLOUD as a good candidate, this is the 9th granule. In this image, our area of interest is close to nadir. This means that the instantaneous field of view over the area of interest cover a smaller area than at the edge of the image.\nWe are looking for the link for direct download access via s3. This is a url but with a prefix s3://. This happens to be the first href link in the metadata.\nFor a single granule we can cut and paste the s3 link. If we have several granules, the s3 links can be extracted with some simple code.\n\ngranule = granules[9]\n\nfor link in granule['links']:\n if link['href'].startswith('s3://'):\n s3_link = link['href']\n \ns3_link\n\n's3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622100001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc'\n\n\n\nGet S3 credentials\nAs with the previous S3 download tutorials we need credentials to access data from s3: access keys and tokens.\n\ns3_credentials = requests.get('https://archive.podaac.earthdata.nasa.gov/s3credentials').json()\n\nEssentially, what we are doing in this step is to “mount” the s3 bucket as a file system. This allows us to treat the S3 bucket in a similar way to a local file system.\n\ns3_fs = s3fs.S3FileSystem(\n key=s3_credentials[\"accessKeyId\"],\n secret=s3_credentials[\"secretAccessKey\"],\n token=s3_credentials[\"sessionToken\"],\n)\n\n\n\nOpen a s3 file\nNow we have the S3FileSystem set up, we can access the granule. xarray cannot open a S3File directly, so we use the open method for the S3FileSystem to open the granule using the endpoint url we extracted from the metadata. We also have to set the mode='rb'. This opens the granule in read-only mode and in byte-mode. Byte-mode is important. By default, open opens a file as text - in this case it would just be a string of characters - and xarray doesn’t know what to do with that.\nWe then pass the S3File object f to xarray.open_dataset. For this dataset, we also have to set decode_cf=False. This switch tells xarray not to use information contained in variable attributes to generate human readable coordinate variables. Normally, this should work for netcdf files but for this particular cloud-hosted dataset, variable attribute data is not in the form expected by xarray. We’ll fix this.\n\nf = s3_fs.open(s3_link, mode='rb')\nmodis_ds = xr.open_dataset(f, decode_cf=False)\n\nIf you click on the Show/Hide Attributes icon (the first document-like icon to the right of coordinate variable metadata) you can see that attributes are one-element arrays containing bytestrings.\n\nmodis_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (nj: 2030, ni: 1354, time: 1)\nCoordinates:\n * time (time) int32 1214042401\nDimensions without coordinates: nj, ni\nData variables:\n lat (nj, ni) float32 ...\n lon (nj, ni) float32 ...\n sea_surface_temperature (time, nj, ni) int16 ...\n sst_dtime (time, nj, ni) int16 ...\n quality_level (time, nj, ni) int8 ...\n sses_bias (time, nj, ni) int8 ...\n sses_standard_deviation (time, nj, ni) int8 ...\n l2p_flags (time, nj, ni) int16 ...\n chlorophyll_a (time, nj, ni) float32 ...\n K_490 (time, nj, ni) int16 ...\n wind_speed (time, nj, ni) int8 ...\n dt_analysis (time, nj, ni) int8 ...\nAttributes: (12/49)\n Conventions: [b'CF-1.7, ACDD-1.3']\n title: [b'MODIS Aqua L2P SST']\n summary: [b'Sea surface temperature retrievals produce...\n references: [b'GHRSST Data Processing Specification v2r5']\n institution: [b'NASA/JPL/OBPG/RSMAS']\n history: [b'MODIS L2P created at JPL PO.DAAC']\n ... ...\n publisher_email: [b'ghrsst-po@nceo.ac.uk']\n processing_level: [b'L2P']\n cdm_data_type: [b'swath']\n startDirection: [b'Ascending']\n endDirection: [b'Descending']\n day_night_flag: [b'Day']xarray.DatasetDimensions:nj: 2030ni: 1354time: 1Coordinates: (1)time(time)int321214042401long_name :[b'reference time of sst file']standard_name :[b'time']units :[b'seconds since 1981-01-01 00:00:00']comment :[b'time of first sensor observation']coverage_content_type :[b'coordinate']array([1214042401], dtype=int32)Data variables: (12)lat(nj, ni)float32...long_name :[b'latitude']standard_name :[b'latitude']units :[b'degrees_north']_FillValue :[-999.]valid_min :[-90.]valid_max :[90.]comment :[b'geographical coordinates, WGS84 projection']coverage_content_type :[b'coordinate'][2748620 values with dtype=float32]lon(nj, ni)float32...long_name :[b'longitude']standard_name :[b'longitude']units :[b'degrees_east']_FillValue :[-999.]valid_min :[-180.]valid_max :[180.]comment :[b'geographical coordinates, WGS84 projection']coverage_content_type :[b'coordinate'][2748620 values with dtype=float32]sea_surface_temperature(time, nj, ni)int16...long_name :[b'sea surface temperature']standard_name :[b'sea_surface_skin_temperature']units :[b'kelvin']_FillValue :[-32767]valid_min :[-1000]valid_max :[10000]comment :[b'sea surface temperature from thermal IR (11 um) channels']scale_factor :[0.005]add_offset :[273.15]source :[b'NASA and University of Miami']coordinates :[b'lon lat']coverage_content_type :[b'physicalMeasurement'][2748620 values with dtype=int16]sst_dtime(time, nj, ni)int16...long_name :[b'time difference from reference time']units :[b'seconds']_FillValue :[-32768]valid_min :[-32767]valid_max :[32767]comment :[b'time plus sst_dtime gives seconds after 00:00:00 UTC January 1, 1981']coordinates :[b'lon lat']coverage_content_type :[b'referenceInformation'][2748620 values with dtype=int16]quality_level(time, nj, ni)int8...long_name :[b'quality level of SST pixel']_FillValue :[-128]valid_min :[0]valid_max :[5]comment :[b'thermal IR SST proximity confidence value; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value']coordinates :[b'lon lat']flag_values :[0 1 2 3 4 5]flag_meanings :[b'no_data bad_data worst_quality low_quality acceptable_quality best_quality']coverage_content_type :[b'qualityInformation'][2748620 values with dtype=int8]sses_bias(time, nj, ni)int8...long_name :[b'SSES bias error based on proximity confidence flags']units :[b'kelvin']_FillValue :[-128]valid_min :[-127]valid_max :[127]comment :[b'thermal IR SST bias error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value']scale_factor :[0.15748031]add_offset :[0.]coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int8]sses_standard_deviation(time, nj, ni)int8...long_name :[b'SSES standard deviation error based on proximity confidence flags']units :[b'kelvin']_FillValue :[-128]valid_min :[-127]valid_max :[127]comment :[b'thermal IR SST standard deviation error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value']scale_factor :[0.07874016]add_offset :[10.]coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int8]l2p_flags(time, nj, ni)int16...long_name :[b'L2P flags']valid_min :[0]valid_max :[16]comment :[b'These flags can be used to further filter data variables']coordinates :[b'lon lat']flag_meanings :[b'microwave land ice lake river']flag_masks :[ 1 2 4 8 16]coverage_content_type :[b'qualityInformation'][2748620 values with dtype=int16]chlorophyll_a(time, nj, ni)float32...long_name :[b'Chlorophyll Concentration, OC3 Algorithm']units :[b'mg m^-3']_FillValue :[-32767.]valid_min :[0.001]valid_max :[100.]comment :[b'non L2P core field']coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=float32]K_490(time, nj, ni)int16...long_name :[b'Diffuse attenuation coefficient at 490 nm (OBPG)']units :[b'm^-1']_FillValue :[-32767]valid_min :[50]valid_max :[30000]comment :[b'non L2P core field']scale_factor :[0.0002]add_offset :[0.]coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int16]wind_speed(time, nj, ni)int8...long_name :[b'10m wind speed']standard_name :[b'wind_speed']units :[b'm s-1']_FillValue :[-128]valid_min :[-127]valid_max :[127]comment :[b'Wind at 10 meters above the sea surface']scale_factor :[0.2]add_offset :[25.]source :[b'TBD. Placeholder. Currently empty']coordinates :[b'lon lat']grid_mapping :[b'TBD']time_offset :[2.]height :[b'10 m']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int8]dt_analysis(time, nj, ni)int8...long_name :[b'deviation from SST reference climatology']units :[b'kelvin']_FillValue :[-128]valid_min :[-127]valid_max :[127]comment :[b'TBD']scale_factor :[0.1]add_offset :[0.]source :[b'TBD. Placeholder. Currently empty']coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int8]Attributes: (49)Conventions :[b'CF-1.7, ACDD-1.3']title :[b'MODIS Aqua L2P SST']summary :[b'Sea surface temperature retrievals produced at the NASA OBPG for the MODIS Aqua sensor. These have been reformatted to GHRSST GDS specifications by the JPL PO.DAAC']references :[b'GHRSST Data Processing Specification v2r5']institution :[b'NASA/JPL/OBPG/RSMAS']history :[b'MODIS L2P created at JPL PO.DAAC']comment :[b'L2P Core without DT analysis or other ancillary fields; Day, Start Node:Ascending, End Node:Descending; WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value; Refined']license :[b'GHRSST and PO.DAAC protocol allow data use as free and open.']id :[b'MODIS_A-JPL-L2P-v2019.0']naming_authority :[b'org.ghrsst']product_version :[b'2019.0']uuid :[b'f6e1f61d-c4a4-4c17-8354-0c15e12d688b']gds_version_id :[b'2.0']netcdf_version_id :[b'4.1']date_created :[b'20200221T085224Z']file_quality_level :[3]spatial_resolution :[b'1km']start_time :[b'20190622T100001Z']time_coverage_start :[b'20190622T100001Z']stop_time :[b'20190622T100459Z']time_coverage_end :[b'20190622T100459Z']northernmost_latitude :[89.9862]southernmost_latitude :[66.2723]easternmost_longitude :[-45.9467]westernmost_longitude :[152.489]source :[b'MODIS sea surface temperature observations for the OBPG']platform :[b'Aqua']sensor :[b'MODIS']metadata_link :[b'http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MODIS_A-JPL-L2P-v2019.0']keywords :[b'Oceans > Ocean Temperature > Sea Surface Temperature']keywords_vocabulary :[b'NASA Global Change Master Directory (GCMD) Science Keywords']standard_name_vocabulary :[b'NetCDF Climate and Forecast (CF) Metadata Convention']geospatial_lat_units :[b'degrees_north']geospatial_lat_resolution :[0.01]geospatial_lon_units :[b'degrees_east']geospatial_lon_resolution :[0.01]acknowledgment :[b'The MODIS L2P sea surface temperature data are sponsored by NASA']creator_name :[b'Ed Armstrong, JPL PO.DAAC']creator_email :[b'edward.m.armstrong@jpl.nasa.gov']creator_url :[b'http://podaac.jpl.nasa.gov']project :[b'Group for High Resolution Sea Surface Temperature']publisher_name :[b'The GHRSST Project Office']publisher_url :[b'http://www.ghrsst.org']publisher_email :[b'ghrsst-po@nceo.ac.uk']processing_level :[b'L2P']cdm_data_type :[b'swath']startDirection :[b'Ascending']endDirection :[b'Descending']day_night_flag :[b'Day']\n\n\nTo fix this, we need to extract array elements as scalars, and convert those scalars from bytestrings to strings. We use the decode method to do this. The bytestrings are encoded as utf-8, which is a unicode character format. This is the default encoding for decode but we’ve included it as an argument to be explicit.\nNot all attributes are bytestrings. Some are floats. Take a look at _FillValue, and valid_min and valid_max. To avoid an error, we use the isinstance function to check if the value of an attributes is type bytes - a bytestring. If it is, then we decode it. If not, we just extract the scalar and do nothing else.\nWe also fix the global attributes.\n\ndef fix_attributes(da):\n '''Decodes bytestring attributes to strings'''\n for attr, value in da.attrs.items():\n if isinstance(value[0], bytes):\n da.attrs[attr] = value[0].decode('utf-8')\n else:\n da.attrs[attr] = value[0]\n return\n\n# Fix variable attributes\nfor var in modis_ds.variables:\n da = modis_ds[var]\n fix_attributes(da)\n \n# Fix global attributes\nfix_attributes(modis_ds)\n\nWith this done, we can use the xarray function decode_cf to convert the attributes.\n\nmodis_ds = xr.decode_cf(modis_ds)\n\n\nmodis_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (nj: 2030, ni: 1354, time: 1)\nCoordinates:\n lat (nj, ni) float32 ...\n lon (nj, ni) float32 ...\n * time (time) datetime64[ns] 2019-06-22T10:00:01\nDimensions without coordinates: nj, ni\nData variables:\n sea_surface_temperature (time, nj, ni) float32 ...\n sst_dtime (time, nj, ni) timedelta64[ns] ...\n quality_level (time, nj, ni) float32 ...\n sses_bias (time, nj, ni) float32 ...\n sses_standard_deviation (time, nj, ni) float32 ...\n l2p_flags (time, nj, ni) int16 ...\n chlorophyll_a (time, nj, ni) float32 ...\n K_490 (time, nj, ni) float32 ...\n wind_speed (time, nj, ni) float32 ...\n dt_analysis (time, nj, ni) float32 ...\nAttributes: (12/49)\n Conventions: CF-1.7, ACDD-1.3\n title: MODIS Aqua L2P SST\n summary: Sea surface temperature retrievals produced a...\n references: GHRSST Data Processing Specification v2r5\n institution: NASA/JPL/OBPG/RSMAS\n history: MODIS L2P created at JPL PO.DAAC\n ... ...\n publisher_email: ghrsst-po@nceo.ac.uk\n processing_level: L2P\n cdm_data_type: swath\n startDirection: Ascending\n endDirection: Descending\n day_night_flag: Dayxarray.DatasetDimensions:nj: 2030ni: 1354time: 1Coordinates: (3)lat(nj, ni)float32...long_name :latitudestandard_name :latitudeunits :degrees_northvalid_min :-90.0valid_max :90.0comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[2748620 values with dtype=float32]lon(nj, ni)float32...long_name :longitudestandard_name :longitudeunits :degrees_eastvalid_min :-180.0valid_max :180.0comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[2748620 values with dtype=float32]time(time)datetime64[ns]2019-06-22T10:00:01long_name :reference time of sst filestandard_name :timecomment :time of first sensor observationcoverage_content_type :coordinatearray(['2019-06-22T10:00:01.000000000'], dtype='datetime64[ns]')Data variables: (10)sea_surface_temperature(time, nj, ni)float32...long_name :sea surface temperaturestandard_name :sea_surface_skin_temperatureunits :kelvinvalid_min :-1000valid_max :10000comment :sea surface temperature from thermal IR (11 um) channelssource :NASA and University of Miamicoverage_content_type :physicalMeasurement[2748620 values with dtype=float32]sst_dtime(time, nj, ni)timedelta64[ns]...long_name :time difference from reference timevalid_min :-32767valid_max :32767comment :time plus sst_dtime gives seconds after 00:00:00 UTC January 1, 1981coverage_content_type :referenceInformation[2748620 values with dtype=timedelta64[ns]]quality_level(time, nj, ni)float32...long_name :quality level of SST pixelvalid_min :0valid_max :5comment :thermal IR SST proximity confidence value; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valueflag_values :0flag_meanings :no_data bad_data worst_quality low_quality acceptable_quality best_qualitycoverage_content_type :qualityInformation[2748620 values with dtype=float32]sses_bias(time, nj, ni)float32...long_name :SSES bias error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :thermal IR SST bias error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]sses_standard_deviation(time, nj, ni)float32...long_name :SSES standard deviation error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :thermal IR SST standard deviation error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]l2p_flags(time, nj, ni)int16...long_name :L2P flagsvalid_min :0valid_max :16comment :These flags can be used to further filter data variablesflag_meanings :microwave land ice lake riverflag_masks :1coverage_content_type :qualityInformation[2748620 values with dtype=int16]chlorophyll_a(time, nj, ni)float32...long_name :Chlorophyll Concentration, OC3 Algorithmunits :mg m^-3valid_min :0.001valid_max :100.0comment :non L2P core fieldcoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]K_490(time, nj, ni)float32...long_name :Diffuse attenuation coefficient at 490 nm (OBPG)units :m^-1valid_min :50valid_max :30000comment :non L2P core fieldcoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]wind_speed(time, nj, ni)float32...long_name :10m wind speedstandard_name :wind_speedunits :m s-1valid_min :-127valid_max :127comment :Wind at 10 meters above the sea surfacesource :TBD. Placeholder. Currently emptygrid_mapping :TBDtime_offset :2.0height :10 mcoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]dt_analysis(time, nj, ni)float32...long_name :deviation from SST reference climatologyunits :kelvinvalid_min :-127valid_max :127comment :TBDsource :TBD. Placeholder. Currently emptycoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]Attributes: (49)Conventions :CF-1.7, ACDD-1.3title :MODIS Aqua L2P SSTsummary :Sea surface temperature retrievals produced at the NASA OBPG for the MODIS Aqua sensor. These have been reformatted to GHRSST GDS specifications by the JPL PO.DAACreferences :GHRSST Data Processing Specification v2r5institution :NASA/JPL/OBPG/RSMAShistory :MODIS L2P created at JPL PO.DAACcomment :L2P Core without DT analysis or other ancillary fields; Day, Start Node:Ascending, End Node:Descending; WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value; Refinedlicense :GHRSST and PO.DAAC protocol allow data use as free and open.id :MODIS_A-JPL-L2P-v2019.0naming_authority :org.ghrsstproduct_version :2019.0uuid :f6e1f61d-c4a4-4c17-8354-0c15e12d688bgds_version_id :2.0netcdf_version_id :4.1date_created :20200221T085224Zfile_quality_level :3spatial_resolution :1kmstart_time :20190622T100001Ztime_coverage_start :20190622T100001Zstop_time :20190622T100459Ztime_coverage_end :20190622T100459Znorthernmost_latitude :89.9862southernmost_latitude :66.2723easternmost_longitude :-45.9467westernmost_longitude :152.489source :MODIS sea surface temperature observations for the OBPGplatform :Aquasensor :MODISmetadata_link :http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MODIS_A-JPL-L2P-v2019.0keywords :Oceans > Ocean Temperature > Sea Surface Temperaturekeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordsstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventiongeospatial_lat_units :degrees_northgeospatial_lat_resolution :0.01geospatial_lon_units :degrees_eastgeospatial_lon_resolution :0.01acknowledgment :The MODIS L2P sea surface temperature data are sponsored by NASAcreator_name :Ed Armstrong, JPL PO.DAACcreator_email :edward.m.armstrong@jpl.nasa.govcreator_url :http://podaac.jpl.nasa.govproject :Group for High Resolution Sea Surface Temperaturepublisher_name :The GHRSST Project Officepublisher_url :http://www.ghrsst.orgpublisher_email :ghrsst-po@nceo.ac.ukprocessing_level :L2Pcdm_data_type :swathstartDirection :AscendingendDirection :Descendingday_night_flag :Day\n\n\nLet’s make a quick plot to take a look at the sea_surface_temperature variable.\n\nmodis_ds.sea_surface_temperature.plot() ;\n\n\n\n\n\n\nPlot MODIS and ICESat-2 data on a map\n\nmap_proj = ccrs.NorthPolarStereo()\n\nfig = plt.figure(figsize=(10,5))\nax = fig.add_subplot(projection=map_proj)\nax.coastlines()\n\n# Plot MODIS sst, save object as sst_img, so we can add colorbar\nsst_img = ax.pcolormesh(modis_ds.lon, modis_ds.lat, modis_ds.sea_surface_temperature[0,:,:], \n vmin=240, vmax=270, # Set max and min values for plotting\n cmap='viridis', shading='auto', # shading='auto' to avoid warning\n transform=ccrs.PlateCarree()) # coords are lat,lon but map if NPS \n\n# Plot IS2 surface height \nis2_img = ax.scatter(is2_ds.longitude, is2_ds.latitude,\n c=is2_ds.height_segment_height, \n vmax=1.5, # Set max height to plot\n cmap='Reds', alpha=0.6, s=2,\n transform=ccrs.PlateCarree())\n\n# Add colorbars\nfig.colorbar(sst_img, label='MODIS SST (K)')\nfig.colorbar(is2_img, label='ATL07 Height (m)')\n\n\n<matplotlib.colorbar.Colorbar at 0x7fb3944adb50>\n\n\n\n\n\n\n\nExtract SST coincident with ICESat-2 track\nThe MODIS SST is swath data, not a regularly-spaced grid of sea surface temperatures. ICESat-2 sea surface heights are irregularly spaced segments along one ground-track traced by the ATLAS instrument on-board ICESat-2. Fortunately, pyresample allows us to resample swath data.\npyresample has many resampling methods. We’re going to use the nearest neighbour resampling method, which is implemented using a k-dimensional tree algorithm or K-d tree. K-d trees are data structures that improve search efficiency for large data sets.\nThe first step is to define the geometry of the ICESat-2 and MODIS data. To do this we use the latitudes and longitudes of the datasets.\n\nis2_geometry = pyresample.SwathDefinition(lons=is2_ds.longitude,\n lats=is2_ds.latitude)\n\n\nmodis_geometry = pyresample.SwathDefinition(lons=modis_ds.lon, lats=modis_ds.lat)\n\nWe then implement the resampling method, passing the two geometries we have defined, the data array we want to resample - in this case sea surface temperature, and a search radius. The resampling method expects a numpy.Array rather than an xarray.DataArray, so we use values to get the data as a numpy.Array.\nWe set the search radius to 1000 m. The MODIS data is nominally 1km spacing.\n\nsearch_radius=1000.\nfill_value = np.nan\nis2_sst = pyresample.kd_tree.resample_nearest(\n modis_geometry,\n modis_ds.sea_surface_temperature.values,\n is2_geometry,\n search_radius,\n fill_value=fill_value\n)\n\n\nis2_sst\n\narray([263.375, 263.375, 263.375, ..., nan, nan, nan],\n dtype=float32)\n\n\n\nis2_ds['sea_surface_temperature'] = xr.DataArray(is2_sst, dims='segment')\nis2_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (segment: 235584)\nDimensions without coordinates: segment\nData variables:\n latitude (segment) float64 82.38 82.38 82.38 ... 72.61 72.61\n longitude (segment) float64 -55.11 -55.11 ... 145.1 145.1\n delta_time (segment) float64 4.642e+07 4.642e+07 ... 4.642e+07\n height_segment_height (segment) float32 nan nan nan ... -0.4335 -0.4463\n sea_surface_temperature (segment) float32 263.4 263.4 263.4 ... nan nan nanxarray.DatasetDimensions:segment: 235584Coordinates: (0)Data variables: (5)latitude(segment)float6482.38 82.38 82.38 ... 72.61 72.61contentType :referenceInformationcoordinates :delta_time longitudedescription :Latitude, WGS84, North=+, Lat of segment centerlong_name :Latitudesource :ATBD, section 4.4standard_name :latitudeunits :degrees_northvalid_max :90.0valid_min :-90.0array([82.38431982, 82.38431982, 82.38431982, ..., 72.60984638,\n 72.60977493, 72.60970985])longitude(segment)float64-55.11 -55.11 ... 145.1 145.1contentType :referenceInformationcoordinates :delta_time latitudedescription :Longitude, WGS84, East=+,Lon of segment centerlong_name :Longitudesource :ATBD, section 4.4standard_name :longitudeunits :degrees_eastvalid_max :180.0valid_min :-180.0array([-55.10896068, -55.10896068, -55.10896068, ..., 145.05396164,\n 145.05392851, 145.05389832])delta_time(segment)float644.642e+07 4.642e+07 ... 4.642e+07CLASS :DIMENSION_SCALENAME :gt1l/sea_ice_segments/delta_timeREFERENCE_LIST :[(<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)]contentType :physicalMeasurementcoordinates :latitude longitudedescription :Number of GPS seconds since the ATLAS SDP epoch. The ATLAS Standard Data Products (SDP) epoch offset is defined within /ancillary_data/atlas_sdp_gps_epoch as the number of GPS seconds between the GPS epoch (1980-01-06T00:00:00.000000Z UTC) and the ATLAS SDP epoch. By adding the offset contained within atlas_sdp_gps_epoch to delta time parameters, the time in gps_seconds relative to the GPS epoch can be computed.long_name :Elapsed GPS secondssource :telemetrystandard_name :timeunits :seconds since 2018-01-01array([46419293.64266939, 46419293.64266939, 46419293.64266939, ...,\n 46419681.87646231, 46419681.87759533, 46419681.87862704])height_segment_height(segment)float32nan nan nan ... -0.4335 -0.4463_FillValue :3.4028235e+38contentType :referenceInformationcoordinates :../delta_time ../latitude ../longitudedescription :Mean height from along-track segment fit detremined by the sea ice algorithm. The sea ice height is relative to the tide-free MSS.long_name :height of segment surfacesource :ATBD, section 4.2.2.4units :metersarray([ nan, nan, nan, ..., -0.46550068,\n -0.43347716, -0.4462675 ], dtype=float32)sea_surface_temperature(segment)float32263.4 263.4 263.4 ... nan nan nanarray([263.375, 263.375, 263.375, ..., nan, nan, nan],\n dtype=float32)Attributes: (0)\n\n\n\n\nPlot SST and Height along track\nThis is a quick plot of the extracted data. We’re using matplotlib so we can use latitude as the x-value:\n\nis2_ds = is2_ds.set_coords(['latitude'])\n\nfig, ax1 = plt.subplots(figsize=(15, 7))\nax1.set_xlim(82.,88.)\nax1.plot(is2_ds.latitude, is2_ds.sea_surface_temperature, \n color='orange', label='SST', zorder=3)\nax1.set_ylabel('SST (K)')\n\nax2 = ax1.twinx()\nax2.plot(is2_ds.latitude, is2_ds.height_segment_height, label='Height')\nax2.set_ylabel('Height (m)')\n\nfig.legend()\n\n<matplotlib.legend.Legend at 0x7fb39fcd8040>" + "objectID": "how-tos/read_data.html", + "href": "how-tos/read_data.html", + "title": "Read Data", + "section": "", + "text": "PythonRMATLAB\n\n\nWe recommend using xarray’s open_dataset\n\n\nimport xarray as xr\nds = xr.open_dataset(filename)\n\n\n\nimport xarray as xr\nds = xr.open_mfdataset(list_of_filenames)\n\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB provides graphical import tools and two programmatic interfaces for interacting with netCDF files.\nThe high-level interface is made of eight high-level functions for getting information about netCDF files and their content, creating variables, writing and reading variables and attributes, and adding schema definitions to netCDF files.\nfilename = \"example.nc\";\nncdisp(filename); % Display file content\nvarname = \"peaks\";\npeaksData = ncread(filename,varname);\nsurf(peaksData);\nThe low-level interface is a thin wrapper around the underlying netCDF C library.\nfilename = \"example.nc\";\nvarname = \"peaks\";\nncID = netcdf.open(filename,\"NC_NOWRITE\");\nvarID = netcdf.inqVarID(ncID,varname);\npeaksData = netcdf.getVar(ncID,varID);\nnetcdf.close(ncID);\nsurf(peaksData);\nNotes\n\nUnlike MATLAB HDF5 interfaces, netCDF interfaces are not cloud enabled (yet).\nAs HDF5 interfaces support reading netCDF 4 files, users must currently use these interfaces for accessing netCDF 4 files stored on the cloud (see tutorial MATLAB Access Single NASA EarthData L2 NetCDF in this Cookbook).\nMATLAB netCDF interfaces support OPeNDAP and byte-range for reading data." }, { - "objectID": "tutorials/earthaccess-demo.html#why", - "href": "tutorials/earthaccess-demo.html#why", - "title": "Introducing NASA earthaccess 🌍", - "section": "Why?", - "text": "Why?\nProgrammatic, Easy, Reproducible.\nThere are many ways to access NASA datasets, we can use the Earthdata search portal. We can use DAAC specific portals or tools. We could even use data.gov! Web portals are great but they are not designed for programmatic access and reproducible workflows. This is extremely important in the age of the cloud and reproducible open science.\nThe good news is that NASA also exposes APIs that allows us to search, transform and access data in a programmatic way. Many of these libraries contain amazing features and some similarities. In this context, earthaccess aims to be a simple library that can deal with the important parts of the metadata so we can access or download data without having to worry if a given dataset is on-prem or in the cloud." + "objectID": "how-tos/read_data.html#netcdf", + "href": "how-tos/read_data.html#netcdf", + "title": "Read Data", + "section": "", + "text": "PythonRMATLAB\n\n\nWe recommend using xarray’s open_dataset\n\n\nimport xarray as xr\nds = xr.open_dataset(filename)\n\n\n\nimport xarray as xr\nds = xr.open_mfdataset(list_of_filenames)\n\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB provides graphical import tools and two programmatic interfaces for interacting with netCDF files.\nThe high-level interface is made of eight high-level functions for getting information about netCDF files and their content, creating variables, writing and reading variables and attributes, and adding schema definitions to netCDF files.\nfilename = \"example.nc\";\nncdisp(filename); % Display file content\nvarname = \"peaks\";\npeaksData = ncread(filename,varname);\nsurf(peaksData);\nThe low-level interface is a thin wrapper around the underlying netCDF C library.\nfilename = \"example.nc\";\nvarname = \"peaks\";\nncID = netcdf.open(filename,\"NC_NOWRITE\");\nvarID = netcdf.inqVarID(ncID,varname);\npeaksData = netcdf.getVar(ncID,varID);\nnetcdf.close(ncID);\nsurf(peaksData);\nNotes\n\nUnlike MATLAB HDF5 interfaces, netCDF interfaces are not cloud enabled (yet).\nAs HDF5 interfaces support reading netCDF 4 files, users must currently use these interfaces for accessing netCDF 4 files stored on the cloud (see tutorial MATLAB Access Single NASA EarthData L2 NetCDF in this Cookbook).\nMATLAB netCDF interfaces support OPeNDAP and byte-range for reading data." }, { - "objectID": "tutorials/earthaccess-demo.html#how", - "href": "tutorials/earthaccess-demo.html#how", - "title": "Introducing NASA earthaccess 🌍", - "section": "How?", - "text": "How?\n\nNote: There are a lot of acronyms that we need to get familiar with before any of this makes sense, here is a brief glossary for NASA Earthdata terms: NASA glossary\n\n\nAuthentication: Before we can use earthaccess we need an account with NASA EDL\nEarthdata Login provides free and immediate access to thousands of EOSDIS data products covering all Earth science disciplines and topic areas for researchers, applied science users, application developers, and the general public.\nOnce we have our NASA EDL login credentials we can start accessing NASA data in a programmatic way.\n\nimport earthaccess\nearthaccess.__version__\n\n'0.3.0'\n\n\n\nfrom earthaccess import Auth, Store, DataCollections, DataGranules\nauth = Auth()\n\n\nAuth()\nearthaccess’s Auth class provides 3 different strategies to authenticate ourselves with NASA EDL.\n\nnetrc: Do we have a .netrc file with our EDL credentials? if so, we can use it with earthaccess. If we don’t have it and want to create one we can, earthaccess allows users to type their credentials and persist them into a .netrc file.\nenvironment: If we have our EDL credentials as environment variables\n\nEDL_USERNAME\nEDL_PASSWORD\n\ninteractive: We will be asked for our EDL credentials with optinal persistance to .netrc\n\nTo persist our credentials to a .netrc file we have to do the following:\nauth.login(strategy=\"interactive\", persist=True)\n\nauth.login(strategy=\"netrc\")\n# are we authenticated?\nprint(auth.authenticated)\n\nYou're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 07/24/2022\nTrue\n\n\n\na = auth.login(strategy=\"environment\")\n# are we authenticated?\nprint(auth.authenticated)\n\nWe are already authenticated with NASA EDL\nTrue" + "objectID": "how-tos/read_data.html#hdf5", + "href": "how-tos/read_data.html#hdf5", + "title": "Read Data", + "section": "HDF5", + "text": "HDF5\n\nPythonRMATLAB\n\n\nFor HDF5 files, there are two methods we can use: xarray’s open_dataset and h5py\nimport xarray as xr\nds = xr.open_dataset(filename,\n group=path_to_H5netCDF_group)\n# add in directions for h5py\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB provides low- and high-level programmatic interfaces for interacting with HDF5 and HDF4 files.\nFocusing on HDF5, which is currently more prevalent, the high-level interface is made of seven high-level functions for getting information about HDF5 files and their content, creating datasets, and reading and writing datasets and attributes.\nfilename = \"example.h5\";\nh5disp(filename); % Display file content\nds = \"/g4/lat\";\nlat = h5read(filename,ds);\nThe low-level interface is a thin wrapper around the underlying HDF5 C library. It is made of more than 300 functions organized in packages (H5F for file management, H5D for dataset management, etc.).\nfilename = \"example.h5\";\nds = \"/g4/lat\";\nfID = H5F.open(filename); \ndsID = H5D.open(fID,ds);\nlat = H5D.read(dsID);\nH5D.close(dsID);\nH5F.close(fID);" }, { - "objectID": "tutorials/earthaccess-demo.html#querying-for-datasets", - "href": "tutorials/earthaccess-demo.html#querying-for-datasets", - "title": "Introducing NASA earthaccess 🌍", - "section": "Querying for datasets", - "text": "Querying for datasets\nThe DataCollections class can query CMR for any collection (dataset) using all of CMR’s Query parameters and has built-in functions to extract useful information from the response.\n\n# The first step is to create a DataCollections query \nQuery = DataCollections()\n\n# Use chain methods to customize our query\nQuery.keyword('elevation').bounding_box(-134.7,58.9,-133.9,59.2).temporal(\"2020-01-01\",\"2020-02-01\")\n\nprint(f'Collections found: {Query.hits()}')\n\n# filtering what UMM fields to print, to see the full record we omit the fields filters\n# meta is always included as \ncollections = Query.fields(['ShortName','Version']).get(5)\n# Inspect some results printing just the ShortName and Abstract\ncollections[0:3]\n\nThe results from a DataCollections and DataGranules query are enhanced python dictionaries, this means that we can access all the keys and values like we usually do with Python dictionaries.\n\ncollections[0][\"umm\"][\"ShortName\"]\n\nThe DataCollections class returns python dictionaries with some handy methods.\ncollection.concept_id() # returns the concept-id, used to search for data granules\ncollection.abstract() # returns the abstract\ncollection.landing_page() # returns the landing page if present in the UMM fields\ncollection.get_data() # returns the portal where data can be accessed.\nThe same results can be obtained using the dict syntax:\ncollection[\"meta\"][\"concept-id\"] # concept-id\ncollection[\"umm\"][\"RelatedUrls\"] # URLs, with GET DATA, LANDING PAGE etc\n\n# We can now search for collections using a pythonic API client for CMR.\nQuery = DataCollections().daac(\"PODAAC\")\n\nprint(f'Collections found: {Query.hits()}')\ncollections = Query.fields(['ShortName']).get(10)\n# Printing the first collection\ncollections[0]\n\n\n# What if we want cloud collections\nQuery = DataCollections().daac(\"PODAAC\").cloud_hosted(True)\n\nprint(f'Collections found: {Query.hits()}')\ncollections = Query.fields(['ShortName']).get(10)\n# Printing 3 collections\ncollections[0]\n\n\n# Printing the concept-id for the first 10 collections\n[collection.concept_id() for collection in collections]" + "objectID": "how-tos/read_data.html#geotiff", + "href": "how-tos/read_data.html#geotiff", + "title": "Read Data", + "section": "GeoTIFF", + "text": "GeoTIFF\n\nPythonRMATLAB\n\n\nFor GeoTIFF files, we recommend using rioxarray’s open_rasterio\nimport rioxarray\nxds = rioxarray.open_rasterio(\"my.tif\")\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB Mapping Toolbox supports interacting with GeoTIFF files through a series of functions for handling raster data (under Raster Data), and the function geotiffread althrough not recommended but for a few edge cases.\n[A,R] = readgeoraster(\"myGeoTiffFile.tif\");\nmapshow(A,R);" }, { - "objectID": "tutorials/earthaccess-demo.html#querying-for-data-files-granules", - "href": "tutorials/earthaccess-demo.html#querying-for-data-files-granules", - "title": "Introducing NASA earthaccess 🌍", - "section": "Querying for data files (granules)", - "text": "Querying for data files (granules)\nThe DataGranules class provides similar functionality as the collection class. To query for granules in a more reliable way concept-id would be the main key. You can search data granules using a short name but that could (more likely will) return different versions of the same data granules.\nIn this example we’re querying for 10 data grnaules from ICESat-2 ATL06 version 005 dataset.\n\nNote: Generally speaking we won’t need authenticated queries unless they are restricted datasets for early adopters.\n\n\n# We build our query\nfrom pprint import pprint\nQuery = DataGranules().short_name('ATL06').version(\"005\").bounding_box(-134.7,58.9,-133.9,59.2)\n# We get 5 metadata records\ngranules = Query.get(5)\ngranules" + "objectID": "how-tos/read_data.html#shapefiles-geojson", + "href": "how-tos/read_data.html#shapefiles-geojson", + "title": "Read Data", + "section": "Shapefiles & GeoJSON", + "text": "Shapefiles & GeoJSON\n\nPythonRMATLAB\n\n\nTo open shapefiles or GeoJSON, we recommend using geopandas’s read_file.\nimport geopandas as gpd\ngdf = gpd.read_file(filename)\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB Mapping Toolbox supports interacting with shapefiles and GeoJSON files through a series of functions for handling vector data (under Vector Data), in particular readgeotable, shapeinfo, shaperead, and shapewrite.\n% Read shapefile of countries specifying the name of the attributes to read.\nS = shaperead(\"Countries.shp\",\"Attributes\",[\"ISO_2DIGIT\",\"CNTRY_NAME\"]);\n\n% Read GeoJSON data from a website and save it in the file storms.geojson.\nwebsave(\"storms.geojson\",\"https://www.spc.noaa.gov/products/outlook/day1otlk_cat.lyr.geojson\");\nGT = readgeotable(\"storms.geojson\");" }, { - "objectID": "tutorials/earthaccess-demo.html#pretty-printing-data-granules", - "href": "tutorials/earthaccess-demo.html#pretty-printing-data-granules", - "title": "Introducing NASA earthaccess 🌍", - "section": "Pretty printing data granules", - "text": "Pretty printing data granules\nSince we are in a notebook we can take advantage of it to see a more user friendly version of the granules with the built-in function display This will render browse image for the granule if available and eventually will have a similar representation as the one from the Earthdata search portal\n\n# printing 2 granules using display\n[display(granule) for granule in granules]\n\n\nSpatiotemporal queries\nOur granules and collection classes accept the same spatial and temporal arguments as CMR so we can search for granules that match spatiotemporal criteria.\n\nQuery = DataGranules().short_name(\"ATL06\").temporal(\"2020-03-01\", \"2020-03-30\").bounding_box(-134.7,58.9,-133.9,59.2).version(\"005\")\n# Always inspects the hits before retrieven the granule metadata, just because it's very verbose.\nprint(f\"Granules found: {Query.hits()}\")\n\n\n# Now we can print some info about these granules using the built-in methods\ngranules = Query.get(5)\ndata_links = [{'links': g.data_links(access=\"on_prem\"), 'size (MB):': g.size()} for g in granules]\ndata_links" + "objectID": "how-tos/read_data.html#csv", + "href": "how-tos/read_data.html#csv", + "title": "Read Data", + "section": "CSV", + "text": "CSV\n\nPythonRMATLAB\n\n\nTo open CSV files, we recommend using pandas’s read_csv.\nimport pandas as pd\ndf = pd.read_csv(filename)\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB supports interacting with CSV files and other tabular file formats through functions readtable, readtimetable, readcell, and readmatrix, as well as their counterparts for writing to files.\n% Read time table and display summary\nTT = readtimetable(\"myTimeTable.csv\");\nsummary(TT)\nAdvanced workflows can be implemented using detectImportOptions and functions referenced in its documentation.\n% Read variables \"TaxiIn\" and \"TaxiOut\" from airlines dataset\nfilename = \"airlinesmall.csv\";\nopts = detectImportOptions(filename);\nopts.SelectedVariableNames = [\"TaxiIn\",\"TaxiOut\"];\nT = readtable(filename, opts);\nNotes\n\nMATLAB provides higher-level mechanisms form importing data files through the function importdata and the Import Tool.\nA general list of functions and workflows for interacting with deliimited and formatted text files is provided here." }, { - "objectID": "tutorials/earthaccess-demo.html#accessing-the-data", - "href": "tutorials/earthaccess-demo.html#accessing-the-data", - "title": "Introducing NASA earthaccess 🌍", - "section": "Accessing the data", - "text": "Accessing the data\nWith earthaccess a researcher can get the files regardless if they are on-prem or cloud based with the same API call, although an important consideration is that if we want to access data in the cloud (direct access) we must run the code in the cloud. This is because some S3 buckets are configured to only allow direct access (s3:// links) if the requester is in the same zone, us-west-2." + "objectID": "how-tos/read_data.html#excel", + "href": "how-tos/read_data.html#excel", + "title": "Read Data", + "section": "Excel", + "text": "Excel\n\nPythonRMATLAB\n\n\nTo open Excel files, we recommend using pandas’s read_excel\nimport pandas as pd\ndf = pd.read_excel(filename)\n\n\nR code coming soon!\n# Coming soon!\n\n\nSee section about CSV files." }, { - "objectID": "tutorials/earthaccess-demo.html#on-prem-access-daac-hosted-data", - "href": "tutorials/earthaccess-demo.html#on-prem-access-daac-hosted-data", - "title": "Introducing NASA earthaccess 🌍", - "section": "On-prem access: DAAC hosted data 📡", - "text": "On-prem access: DAAC hosted data 📡\nThe Store() class will allow us to download or access our data and needs to be instantiated with our auth instance.\n\nstore = Store(auth)\n\nFor this example we are going to use a PODAAC dataset SMAP_JPL_L3_SSS_CAP_8DAY-RUNNINGMEAN_V5 which we previously queried (see querying for datasets) and got the concept id: C1972955240-PODAAC\n\nQuery = DataGranules().concept_id(\"C1972955240-PODAAC\").bounding_box(-134.7,54.9,-100.9,69.2)\nprint(f\"Granule hits: {Query.hits()}\")\n# getting more than 6,000 metadata records for demo purposes is going to slow us down a bit so let's get only a few\ngranules = Query.get(10)\n# Does this granule belong to a cloud-based collection?\ngranules[0].cloud_hosted\n\n\nFinally! let’s get the data\nThe Store class accepts the results from a DataGranules() query or it can also accept a list of URLs for the data files. In the second case we’ll have to specify the DAAC since it cannot infer which credentials to use solely on the URL.\n\n%%time\nfiles = store.get(granules[0:4], \"./data/C1972955240-PODAAC/\")" + "objectID": "how-tos/read_data.html#mat-files-.mat", + "href": "how-tos/read_data.html#mat-files-.mat", + "title": "Read Data", + "section": "MAT-Files (.mat)", + "text": "MAT-Files (.mat)\n\nPythonRMATLAB\n\n\n#coming soon! scipy & .mat\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB can interact with MAT-Files (.mat) using functions load, save, and through matfile objects.\nNotes\n\nMATLAB provides higher-level mechanisms form importing data files through the function importdata and the Import Tool.\nMultiple versions of the MAT-File format have been released. The latest, MAT-File v7.3, is based on the HDF5 file format." }, { - "objectID": "tutorials/earthaccess-demo.html#accessing-the-data-in-the-cloud", - "href": "tutorials/earthaccess-demo.html#accessing-the-data-in-the-cloud", - "title": "Introducing NASA earthaccess 🌍", - "section": "Accessing the data in the cloud ☁️", - "text": "Accessing the data in the cloud ☁️\n\nWith earthaccess a researcher can get the files regardless if they are on-prem or cloud based with the same API call, although an important consideration is that if we want to access data in the cloud we must run the code in the cloud. This is because some S3 buckets are configured to only allow direct access (s3:// links) if the requester is in the same zone, us-west-2.\nSame API, just a different place, in this case the concept-id for the same dataset is C2208422957-POCLOUD > Note: The concept-id changed even though is the same dataset.\n\n\nQuery = DataGranules().concept_id(\"C2208422957-POCLOUD\").bounding_box(-134.7,54.9,-100.9,69.2)\nprint(f\"Granule hits: {Query.hits()}\")\ncloud_granules = Query.get(10)\n# is this a cloud hosted data granule?\ncloud_granules[0].cloud_hosted\n\n\n# Let's pretty print this\ncloud_granules[0]\n\n\n%%time\n# If we get an error with direct_access=True, most likely is because\n# we are running this code outside the us-west-2 region.\ntry:\n files = store.get(cloud_granules[0:4], local_path=\"./data/demo-POCLOUD\")\nexcept Exception as e:\n print(f\"Error: {e}, we are probably not using this code in the Amazon cloud. Trying external links...\")\n # There is hope, even if we are not in the Amazon cloud we can still get the data\n files = store.get(cloud_granules[0:4], access=\"external\", local_path=\"./data/demo-POCLOUD\")" + "objectID": "how-tos/read_data.html#text-data-files", + "href": "how-tos/read_data.html#text-data-files", + "title": "Read Data", + "section": "Text Data Files", + "text": "Text Data Files\n\nPythonRMATLAB\n\n\n# Coming soon!\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB provides a series of functions for interacting with text files. In addition to functions listed in sections about CSV and Binary files, MATLAB provides functions fileread for reading the content of a whole file as text, and readlines and writelines for reading/writing lines to text files.\nNotes\n\nMATLAB provides higher-level mechanisms form importing data files through the function importdata and the Import Tool.\nMATLAB patterns and regular expressions are useful for processing unstructured text data." }, { - "objectID": "tutorials/earthaccess-demo.html#cloud-access-part-ii-streaming-data", - "href": "tutorials/earthaccess-demo.html#cloud-access-part-ii-streaming-data", - "title": "Introducing NASA earthaccess 🌍", - "section": "☁️ Cloud Access Part II: streaming data", - "text": "☁️ Cloud Access Part II: streaming data\nBeing in the cloud allows us to stream data as if we were using it locally. Pairing gridded datasets on S3 and xarray isa very useful patter when we deal with a lot of data.\n\nRecommended read: Skip the download! Stream NASA data directly into Python objects\n\n\nimport warnings\nwarnings.filterwarnings('ignore')\nwarnings.simplefilter('ignore')\nimport xarray as xr\n# data_links\nhttps_links = []\ns3_links = []\n\nfs = store.get_s3fs_session('POCLOUD')\n\nfor granule in cloud_granules:\n https_links.extend(granule.data_links(access=\"on_prem\"))\n s3_links.extend(granule.data_links(access=\"direct\"))\ns3_links\n\n\n%%time\n\nimport xarray as xr\n\ntry:\n files = store.open(s3_links, provider=\"POCLOUD\")\n\n ds_L3 = xr.open_mfdataset(\n files,\n combine='nested',\n concat_dim='time',\n decode_cf=True,\n coords='minimal',\n chunks={'time': 1}\n )\n ds_L3\nexcept Exception as e:\n pass\n # print(e)" + "objectID": "how-tos/read_data.html#binary-files", + "href": "how-tos/read_data.html#binary-files", + "title": "Read Data", + "section": "Binary Files", + "text": "Binary Files\n\nPythonRMATLAB\n\n\nTo open binary files, we recommend using numpy’s from_file. You will need to know the dimensions and data type of the binary file and if there is an offset or header information in the binary file.\nimport numpy as np\narr = np.from_file(filepath, dtype=data_type).reshape(nrows, ncols)\n\n\nR code coming soon!\n# Coming soon!\n\n\nMATLAB supports interacting with binary files through low-level IO functions fopen, fclose, ferror, fseek, ftell, feof, fscanf, fprintf, fread, fwrite, and frewind.\nNotes\n\nMATLAB provides higher-level mechanisms form importing data files through the function importdata and the Import Tool.\nIt is rare that ones must implement custom binary data file readers/writers, because MATLAB and its toolboxes provide functions for reading and writing a large number of data file formats." }, { - "objectID": "tutorials/earthaccess-demo.html#now-to-the-important-science", - "href": "tutorials/earthaccess-demo.html#now-to-the-important-science", - "title": "Introducing NASA earthaccess 🌍", - "section": "Now to the important science! 🚀", - "text": "Now to the important science! 🚀\n\nRecap\n\nfrom earthaccess import Auth, DataGranules, Store\n\n# first we authenticate with NASA EDL\nauth = Auth().login(strategy=\"netrc\")\n\n# Then we build a Query with spatiotemporal parameters\nGranuleQuery = DataGranules().concept_id(\"C1575731655-LPDAAC_ECS\").bounding_box(-134.7,58.9,-133.9,59.2)\n\n# We get the metadata records from CMR\ngranules = GranuleQuery.get()\n\n# Now it{s time to download (or open) our data granules list with get()\nfiles = Store(auth).get(granules, local_path='./data')\n\n# Now to the important science!\n\n\nRelated links\nGithub repository: https://github.com/nsidc/earthaccess\nCMR API documentation: https://cmr.earthaccess.nasa.gov/search/site/docs/search/api.html\nEDL API documentation: https://urs.earthaccess.nasa.gov/\nNASA OpenScapes: https://nasa-openscapes.github.io/earthaccess-cloud-cookbook/\nNSIDC: https://nsidc.org\nContact: luis.lopez@nsidc.org" + "objectID": "how-tos/access-data/access-local-python.html", + "href": "how-tos/access-data/access-local-python.html", + "title": "How do I access Cloud Data from my Local Machine in Python?", + "section": "", + "text": "When you have found the data you want to use, you have two options. You can download the data to work locally, or access the data directly to work in the cloud. This second way of working is called “Direct Cloud Access” or simply, “Direct Access”.\nThis page covers downloading data from Earthdata Cloud to your local machine using the earthaccess library." }, { - "objectID": "external/zarr-eosdis-store.html", - "href": "external/zarr-eosdis-store.html", - "title": "Zarr Example", + "objectID": "how-tos/access-data/access-local-python.html#downloading-data", + "href": "how-tos/access-data/access-local-python.html#downloading-data", + "title": "How do I access Cloud Data from my Local Machine in Python?", "section": "", - "text": "imported on: 2023-05-30\n\nThis notebook is from NASA’s Zarr EOSDIS store notebook\n\n\nThe original source for this document is https://github.com/nasa/zarr-eosdis-store\n\n\nzarr-eosdis-store example\nInstall dependencies\n\nimport sys\n\n# zarr and zarr-eosdis-store, the main libraries being demoed\n!{sys.executable} -m pip install zarr zarr-eosdis-store\n\n# Notebook-specific libraries\n!{sys.executable} -m pip install matplotlib\n\nImportant: To run this, you must first create an Earthdata Login account (https://urs.earthdata.nasa.gov) and place your credentials in ~/.netrc e.g.:\n machine urs.earthdata.nasa.gov login YOUR_USER password YOUR_PASSWORD\nNever share or commit your password / .netrc file!\nBasic usage. After these lines, we work with ds as though it were a normal Zarr dataset\n\nimport zarr\nfrom eosdis_store import EosdisStore\n\nurl = 'https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20210715090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc'\n\nds = zarr.open(EosdisStore(url))\n\nView the file’s variable structure\n\nprint(ds.tree())\n\n/\n ├── analysed_sst (1, 17999, 36000) int16\n ├── analysis_error (1, 17999, 36000) int16\n ├── dt_1km_data (1, 17999, 36000) int16\n ├── lat (17999,) float32\n ├── lon (36000,) float32\n ├── mask (1, 17999, 36000) int16\n ├── sea_ice_fraction (1, 17999, 36000) int16\n ├── sst_anomaly (1, 17999, 36000) int16\n └── time (1,) int32\n\n\nFetch the latitude and longitude arrays and determine start and end indices for our area of interest. In this case, we’re looking at the Great Lakes, which have a nice, recognizeable shape. Latitudes 41 to 49, longitudes -93 to 76.\n\nlats = ds['lat'][:]\nlons = ds['lon'][:]\nlat_range = slice(lats.searchsorted(41), lats.searchsorted(49))\nlon_range = slice(lons.searchsorted(-93), lons.searchsorted(-76))\n\nGet the analysed sea surface temperature variable over our area of interest and apply scale factor and offset from the file metadata. In a future release, scale factor and add offset will be automatically applied.\n\nvar = ds['analysed_sst']\nanalysed_sst = var[0, lat_range, lon_range] * var.attrs['scale_factor'] + var.attrs['add_offset']\n\nDraw a pretty picture\n\nfrom matplotlib import pyplot as plt\n\nplt.rcParams[\"figure.figsize\"] = [16, 8]\nplt.imshow(analysed_sst[::-1, :])\nNone\n\n\n\n\nIn a dozen lines of code and a few seconds, we have managed to fetch and visualize the 3.2 megabyte we needed from a 732 megabyte file using the original archive URL and no processing services" + "text": "When you have found the data you want to use, you have two options. You can download the data to work locally, or access the data directly to work in the cloud. This second way of working is called “Direct Cloud Access” or simply, “Direct Access”.\nThis page covers downloading data from Earthdata Cloud to your local machine using the earthaccess library." }, { - "objectID": "external/appeears_csv_cloud_access.html", - "href": "external/appeears_csv_cloud_access.html", - "title": "How to work with AppEEARS Point Sample CSV outputs", + "objectID": "how-tos/access-data/access-local-python.html#earthaccess", + "href": "how-tos/access-data/access-local-python.html#earthaccess", + "title": "How do I access Cloud Data from my Local Machine in Python?", + "section": "earthaccess", + "text": "earthaccess\nWe can use the earthaccess python library to grab the file URLs and then access them with the xarray library.\n\n#Import packages\nimport earthaccess\nimport xarray as xr\n\n\n#Authentication with Earthdata Login\nauth = earthaccess.login(strategy=\"netrc\")\n\nYou're now authenticated with NASA Earthdata Login\nUsing token with expiration date: 10/30/2023\nUsing .netrc file for EDL\n\n\n\n#Access land ice height from ATLAS/ICESat-2 V005 (10.5067/ATLAS/ATL06.005), searching for data over western Greenland coast over two weeks in July 2022. The data are provided as HDF5 granules (files) that span about 1/14th of an orbit.\n\nresults = earthaccess.search_data(short_name=\"ATL06\",\n version=\"005\",\n cloud_hosted=True,\n temporal = (\"2022-07-17\",\"2022-07-31\"),\n bounding_box = (-51.96423,68.10554,-48.71969,70.70529))\n\nGranules found: 5\n\n\n\nfiles = earthaccess.download(results, \"./local_folder\")\n\ndownloaded_files = []\nfor f in files: \n file = \"local_folder/\"+ f\n downloaded_files.append(file)\n\n Getting 5 granules, approx download size: 0.0 GB\nFile ATL06_20220718010029_03991603_005_01.h5 already downloaded\nFile ATL06_20220718134522_04071605_005_01.h5 already downloaded\nFile ATL06_20220722005209_04601603_005_01.h5 already downloaded\nFile ATL06_20220726004352_05211603_005_01.h5 already downloaded\nFile ATL06_20220722133704_04681605_005_01.h5 already downloaded\n\n\n\n\n\n\n\n\n\n\n\n['local_folder/ATL06_20220718010029_03991603_005_01.h5',\n 'local_folder/ATL06_20220718134522_04071605_005_01.h5',\n 'local_folder/ATL06_20220722005209_04601603_005_01.h5',\n 'local_folder/ATL06_20220722133704_04681605_005_01.h5',\n 'local_folder/ATL06_20220726004352_05211603_005_01.h5']\n\n\n\n#Use xarray to load the data as a multifile dataset for a single group in the HDF5 file, in this case land ice segments:\nds = xr.open_mfdataset(downloaded_files, group='/gt1l/land_ice_segments', engine='h5netcdf')\nds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (delta_time: 241711)\nCoordinates:\n * delta_time (delta_time) datetime64[ns] 2022-07-18T01:00:46.67...\n latitude (delta_time) float64 dask.array<chunksize=(78325,), meta=np.ndarray>\n longitude (delta_time) float64 dask.array<chunksize=(78325,), meta=np.ndarray>\nData variables:\n atl06_quality_summary (delta_time) int8 dask.array<chunksize=(78325,), meta=np.ndarray>\n h_li (delta_time) float32 dask.array<chunksize=(78325,), meta=np.ndarray>\n h_li_sigma (delta_time) float32 dask.array<chunksize=(78325,), meta=np.ndarray>\n segment_id (delta_time) float64 dask.array<chunksize=(78325,), meta=np.ndarray>\n sigma_geo_h (delta_time) float32 dask.array<chunksize=(78325,), meta=np.ndarray>\nAttributes:\n Description: The land_ice_height group contains the primary set of deriv...\n data_rate: Data within this group are sparse. Data values are provide...xarray.DatasetDimensions:delta_time: 241711Coordinates: (3)delta_time(delta_time)datetime64[ns]2022-07-18T01:00:46.678760592 .....contentType :referenceInformationdescription :Number of GPS seconds since the ATLAS SDP epoch. The ATLAS Standard Data Products (SDP) epoch offset is defined within /ancillary_data/atlas_sdp_gps_epoch as the number of GPS seconds between the GPS epoch (1980-01-06T00:00:00.000000Z UTC) and the ATLAS SDP epoch. By adding the offset contained within atlas_sdp_gps_epoch to delta time parameters, the time in gps_seconds relative to the GPS epoch can be computed.long_name :Elapsed GPS secondssource :section 4.4standard_name :timearray(['2022-07-18T01:00:46.678760592', '2022-07-18T01:00:46.681322640',\n '2022-07-18T01:00:46.684008720', ..., '2022-07-26T00:49:18.826637808',\n '2022-07-26T00:49:18.829449568', '2022-07-26T00:49:18.832263232'],\n dtype='datetime64[ns]')latitude(delta_time)float64dask.array<chunksize=(78325,), meta=np.ndarray>contentType :physicalMeasurementdescription :Latitude of segment center, WGS84, North=+,long_name :Latitudesource :section 3.10standard_name :latitudeunits :degrees_northvalid_max :90.0valid_min :-90.0\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n1.84 MiB\n611.91 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat64 numpy.ndarray\n\n\n\n\n\n\n\n\nlongitude\n\n\n(delta_time)\n\n\nfloat64\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nphysicalMeasurement\n\ndescription :\n\nLongitude of segment center, , WGS84, East=+\n\nlong_name :\n\nLongitude\n\nsource :\n\nsection 3.10\n\nstandard_name :\n\nlongitude\n\nunits :\n\ndegrees_east\n\nvalid_max :\n\n180.0\n\nvalid_min :\n\n-180.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n1.84 MiB\n611.91 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat64 numpy.ndarray\n\n\n\n\n\n\n\n\nData variables: (5)\n\n\n\n\n\natl06_quality_summary\n\n\n(delta_time)\n\n\nint8\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nqualityInformation\n\ndescription :\n\nThe ATL06_quality_summary parameter indicates the best-quality subset of all ATL06 data. A zero in this parameter implies that no data-quality tests have found a problem with the segment, a one implies that some potential problem has been found. Users who select only segments with zero values for this flag can be relatively certain of obtaining high-quality data, but will likely miss a significant fraction of usable data, particularly in cloudy, rough, or low-surface-reflectance conditions.\n\nflag_meanings :\n\nbest_quality potential_problem\n\nflag_values :\n\n[0 1]\n\nlong_name :\n\nATL06_Quality_Summary\n\nsource :\n\nsection 4.3\n\nunits :\n\n1\n\nvalid_max :\n\n1\n\nvalid_min :\n\n0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n236.05 kiB\n76.49 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nint8 numpy.ndarray\n\n\n\n\n\n\n\n\n\nh_li\n\n\n(delta_time)\n\n\nfloat32\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nphysicalMeasurement\n\ndescription :\n\nStandard land-ice segment height determined by land ice algorithm, corrected for first-photon bias, representing the median- based height of the selected PEs\n\nlong_name :\n\nLand Ice height\n\nsource :\n\nsection 4.4\n\nunits :\n\nmeters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.92 MiB\n305.96 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nh_li_sigma\n\n\n(delta_time)\n\n\nfloat32\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nqualityInformation\n\ndescription :\n\nPropagated error due to sampling error and FPB correction from the land ice algorithm\n\nlong_name :\n\nExpected RMS segment misfit\n\nsource :\n\nsection 4.4\n\nunits :\n\nmeters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.92 MiB\n305.96 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nsegment_id\n\n\n(delta_time)\n\n\nfloat64\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nreferenceInformation\n\ndescription :\n\nSegment number, counting from the equator. Equal to the segment_id for the second of the two 20m ATL03 segments included in the 40m ATL06 segment\n\nlong_name :\n\nReference Point, m\n\nsource :\n\nsection 3.1.2.1\n\nunits :\n\n1\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n1.84 MiB\n611.91 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat64 numpy.ndarray\n\n\n\n\n\n\n\n\n\nsigma_geo_h\n\n\n(delta_time)\n\n\nfloat32\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nqualityInformation\n\ndescription :\n\nTotal vertical geolocation error due to PPD and POD, including the effects of horizontal geolocation error on the segment vertical error.\n\nlong_name :\n\nVertical Geolocation Error\n\nsource :\n\nATBD Section 3.10\n\nunits :\n\nmeters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.92 MiB\n305.96 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nIndexes: (1)delta_timePandasIndexPandasIndex(DatetimeIndex(['2022-07-18 01:00:46.678760592',\n '2022-07-18 01:00:46.681322640',\n '2022-07-18 01:00:46.684008720',\n '2022-07-18 01:00:46.686753504',\n '2022-07-18 01:00:46.689526560',\n '2022-07-18 01:00:46.692315280',\n '2022-07-18 01:00:46.695049040',\n '2022-07-18 01:00:46.700724096',\n '2022-07-18 01:00:46.703545872',\n '2022-07-18 01:00:46.706366832',\n ...\n '2022-07-26 00:49:18.806914512',\n '2022-07-26 00:49:18.809737328',\n '2022-07-26 00:49:18.812559600',\n '2022-07-26 00:49:18.815380608',\n '2022-07-26 00:49:18.818200224',\n '2022-07-26 00:49:18.821015744',\n '2022-07-26 00:49:18.823827088',\n '2022-07-26 00:49:18.826637808',\n '2022-07-26 00:49:18.829449568',\n '2022-07-26 00:49:18.832263232'],\n dtype='datetime64[ns]', name='delta_time', length=241711, freq=None))Attributes: (2)Description :The land_ice_height group contains the primary set of derived ATL06 products. This includes geolocation, height, and standard error and quality measures for each segment. This group is sparse, meaning that parameters are provided only for pairs of segments for which at least one beam has a valid surface-height measurement.data_rate :Data within this group are sparse. Data values are provided only for those ICESat-2 20m segments where at least one beam has a valid land ice height measurement." + }, + { + "objectID": "how-tos/access-data/access-cloud-python.html", + "href": "how-tos/access-data/access-cloud-python.html", + "title": "How do I access data stored in Earthdata Cloud in Python?", "section": "", - "text": "This tutorial demonstrates how to access AppEEARS point sample Comma-Separated Values (CSV) outputs direcly from the cloud. NASA’s Application for Extracting and Exploring Analysis Ready Samples (AρρEEARS) has been migrated to NASA’s Earthdata Cloud space located in AWS us-west 2. This enables the user working in the cloud instance deployed in AWS us-west 2 to access outputs direcly in the cloud using S3 link returned in the location header of the response. In this tutorial, we will walk through the process of submitting a point sample and accessing a CSV outputs from AppEEARS.\nRequirements\n- Earthdata Login Authentication is required to access AppEEARS API and AppEEARS outpurs direcrly from an Amazon AWS bucket. See Requirements section in README.md.\nLearning Objectives\n- Learn how to access AppEEARS point sample CSV outputs direcly from the cloud.\nTutorial Outline\n1. Setting up\n2. Submit a point request\n3. Extract the S3 links to data in S3\n4. Create a boto3 Refreshable Session\n5. Direct S3 access of CSV output\n6. Quality Filtering\n7. Explore the LST time series" + "text": "When you have found the data you want to use, you have two options. You can download the data to work locally, or access the data directly to work in the cloud. This second way of working is called “Direct Cloud Access” or simply, “Direct Access”. Your compute instance needs to be in Amazon Web Services (AWS) Region us-west-2 in order for the code to run successfully. We authenticate using a netrc file and an Earthdata Login, see the appendix for more information on Earthdata Login and netrc setup." }, { - "objectID": "external/appeears_csv_cloud_access.html#setting-up", - "href": "external/appeears_csv_cloud_access.html#setting-up", - "title": "How to work with AppEEARS Point Sample CSV outputs", - "section": "1. Setting up", - "text": "1. Setting up\nImport required packages.\n\nimport requests\nimport numpy\nimport datetime\nimport os\nimport requests\nfrom netrc import netrc\nfrom pathlib import Path\nfrom datetime import datetime, timezone\nfrom botocore.client import Config\nimport pandas\nimport time\nimport warnings\nimport sys\nsys.path.append('../Python/modules/')\nimport aws_session\nwarnings.filterwarnings('ignore')\n\nTo successfully run this tutorial, it is required to create a .netrc file in your home directory. The function _validate_netrc defined in aws_session checks if a properly formatted netrc file exists in your home directory. If the netrc file does not exist, it will prompt you for your Earthdata Login username and password and will create a netrc file. Please see the Prerequisites section in README.md.\n\n# validate if netrc file is present else create one via the user / password prompt for the urs.earthdata.nasa.gov\naws_session._validate_netrc()\n\nINFO : netrc file is setup for urs.earthdata.nasa.gov already ..." + "objectID": "how-tos/access-data/access-cloud-python.html#direct-access", + "href": "how-tos/access-data/access-cloud-python.html#direct-access", + "title": "How do I access data stored in Earthdata Cloud in Python?", + "section": "", + "text": "When you have found the data you want to use, you have two options. You can download the data to work locally, or access the data directly to work in the cloud. This second way of working is called “Direct Cloud Access” or simply, “Direct Access”. Your compute instance needs to be in Amazon Web Services (AWS) Region us-west-2 in order for the code to run successfully. We authenticate using a netrc file and an Earthdata Login, see the appendix for more information on Earthdata Login and netrc setup." }, { - "objectID": "external/appeears_csv_cloud_access.html#submit-a-point-request-in-appeears", - "href": "external/appeears_csv_cloud_access.html#submit-a-point-request-in-appeears", - "title": "How to work with AppEEARS Point Sample CSV outputs", - "section": "2. Submit a point request in AppEEARS", - "text": "2. Submit a point request in AppEEARS\nIn this step, we are going to submit a point request. You can also submit this same request to the AppEEARS Graphic User Interface (GUI) by uploading the JSON file provided in the repository (AppEEARS-Data-Resources/Data/point-request.json). If you have completed the request, save your task_id to a variable, skip this step, and move to the next step of tutorial.\nAssign the AρρEEARS API endpoint to a variable.\n\nappeears_API_endpoint = 'https://appeears.earthdatacloud.nasa.gov/api/'\n\nA Bearer Token is needed to submit requests to the AppEEARS API. To generated a token, a POST request containing Earthdata Login credentials stored in .netrc file is submitted to the login service from the AppEEARS API.\n\nurs = 'urs.earthdata.nasa.gov'\n\n\ntoken_response = requests.post('{}login'.format(appeears_API_endpoint), auth = (netrc().authenticators(urs)[0],netrc().authenticators(urs)[2])).json() # Insert API URL, call login service, provide credentials & return json \ntoken_response\n\n{'token_type': 'Bearer',\n 'token': 'BL2uWT50_qVKOTtcDr2yigaNm-83GN0A1W8lYmiihzSVl2BIoHXnESxNmAMjNWDIGyIwsZusAtcoaxYSpmpl4A',\n 'expiration': '2023-03-15T15:13:36Z'}\n\n\nWe’ll save the token to as an Authorization object we can pass in the header of any request made to the AppEEARS API.\n\ntoken = token_response['token'] # Save login token to a variable\nhead = {'Authorization': 'Bearer {}'.format(token)} # Create a header to store token information, needed to submit a request\n\nThe payload for an AppEEARS API request is a JSON object. Here, we compile the request JSON object with the required parameters. The coordinates (i.e., lat/lon pair) for where the LP DAAC is located are used as our point location. The data layers LST_Night_1km and LST_Day_1km from MOD11A1 version 6.1 are selected below.\n\ntask_name = \"EROS\"\ntask_type = 'point' # Type of task, area or point\nstartDate = '01-01-2015' # Start of the date range for which to extract data: MM-DD-YYYY\nendDate = '03-01-2023' # End of the date range for which to extract data: MM-DD-YYYY\nlat = [43.736038]\nlon = [-96.62139]\nprodLayer = [\n {\"layer\": \"LST_Day_1km\", \"product\": \"MOD11A1.061\"}, \n {\"layer\": \"LST_Night_1km\", \"product\": \"MOD11A1.061\"}\n]\n\n\ntask = {\n 'task_type': task_type,\n 'task_name': task_name,\n 'params': {\n 'dates': [\n {\n 'startDate': startDate,\n 'endDate': endDate\n }],\n 'layers': prodLayer,\n 'coordinates': [\n {\n 'latitude': lat[0],\n 'longitude': lon[0],\n 'id': \"EROS\"\n \n }]\n \n }\n}\n\nNext, submit the AppEEARS request using post function from requests library.\n\ntask_response = requests.post('{}task'.format(appeears_API_endpoint), json=task, headers=head).json() # Post json to the API task service, return response as json\ntask_response\n\n{'task_id': '9b2f9a77-da1a-41ac-ba47-baeb577ad99f', 'status': 'pending'}\n\n\nSave the task_id and wait until your request is processed and complete.\n\ntask_id = task_response['task_id']\ntask_id\n\n'9b2f9a77-da1a-41ac-ba47-baeb577ad99f'\n\n\n\n# Ping API until request is complete, then continue to Section 3\nwhile requests.get('{}task/{}'.format(appeears_API_endpoint, task_id), headers=head).json()['status'] != 'done':\n print(requests.get('{}task/{}'.format(appeears_API_endpoint, task_id), headers=head).json()['status'])\n time.sleep(60)\nprint(requests.get('{}task/{}'.format(appeears_API_endpoint, task_id), headers=head).json()['status'])\n\nqueued\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\ndone" + "objectID": "how-tos/access-data/access-cloud-python.html#python", + "href": "how-tos/access-data/access-cloud-python.html#python", + "title": "How do I access data stored in Earthdata Cloud in Python?", + "section": "Python", + "text": "Python\nWe can use the earthaccess python library to grab the file URLs and then access them with the xarray library.\n\n#Import packages\nimport earthaccess\nimport xarray as xr\n\n\n#Authentication with Earthdata Login\nauth = earthaccess.login(strategy=\"netrc\")\n\n\n#Access land ice height from ATLAS/ICESat-2 V005 (10.5067/ATLAS/ATL06.005), searching for data over western Greenland coast over two weeks in July 2022. The data are provided as HDF5 granules (files) that span about 1/14th of an orbit.\n\nresults = earthaccess.search_data(short_name=\"ATL06\",\n version=\"006\",\n cloud_hosted=True,\n temporal = (\"2022-07-17\",\"2022-07-31\"),\n bounding_box = (-51.96423,68.10554,-48.71969,70.70529))\n\nGranules found: 5\n\n\n\n#Use xarray to load the data as a multifile dataset for a single group in the HDF5 file, in this case land ice segments:\nds = xr.open_mfdataset(earthaccess.open(results), group='/gt1l/land_ice_segments')\nds\n\n Opening 5 granules, approx size: 0.0 GB\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (delta_time: 241711)\nCoordinates:\n * delta_time (delta_time) datetime64[ns] 2022-07-18T01:00:46.67...\n latitude (delta_time) float64 dask.array<chunksize=(78325,), meta=np.ndarray>\n longitude (delta_time) float64 dask.array<chunksize=(78325,), meta=np.ndarray>\nData variables:\n atl06_quality_summary (delta_time) int8 dask.array<chunksize=(78325,), meta=np.ndarray>\n h_li (delta_time) float32 dask.array<chunksize=(78325,), meta=np.ndarray>\n h_li_sigma (delta_time) float32 dask.array<chunksize=(78325,), meta=np.ndarray>\n segment_id (delta_time) float64 dask.array<chunksize=(78325,), meta=np.ndarray>\n sigma_geo_h (delta_time) float32 dask.array<chunksize=(78325,), meta=np.ndarray>\nAttributes:\n Description: The land_ice_height group contains the primary set of deriv...\n data_rate: Data within this group are sparse. Data values are provide...xarray.DatasetDimensions:delta_time: 241711Coordinates: (3)delta_time(delta_time)datetime64[ns]2022-07-18T01:00:46.678760592 .....contentType :referenceInformationdescription :Number of GPS seconds since the ATLAS SDP epoch. The ATLAS Standard Data Products (SDP) epoch offset is defined within /ancillary_data/atlas_sdp_gps_epoch as the number of GPS seconds between the GPS epoch (1980-01-06T00:00:00.000000Z UTC) and the ATLAS SDP epoch. By adding the offset contained within atlas_sdp_gps_epoch to delta time parameters, the time in gps_seconds relative to the GPS epoch can be computed.long_name :Elapsed GPS secondssource :section 4.4standard_name :timearray(['2022-07-18T01:00:46.678760592', '2022-07-18T01:00:46.681322640',\n '2022-07-18T01:00:46.684008720', ..., '2022-07-26T00:49:18.826637808',\n '2022-07-26T00:49:18.829449568', '2022-07-26T00:49:18.832263232'],\n dtype='datetime64[ns]')latitude(delta_time)float64dask.array<chunksize=(78325,), meta=np.ndarray>contentType :physicalMeasurementdescription :Latitude of segment center, WGS84, North=+,long_name :Latitudesource :section 3.10standard_name :latitudeunits :degrees_northvalid_max :90.0valid_min :-90.0\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n1.84 MiB\n611.91 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat64 numpy.ndarray\n\n\n\n\n\n\n\n\nlongitude\n\n\n(delta_time)\n\n\nfloat64\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nphysicalMeasurement\n\ndescription :\n\nLongitude of segment center, , WGS84, East=+\n\nlong_name :\n\nLongitude\n\nsource :\n\nsection 3.10\n\nstandard_name :\n\nlongitude\n\nunits :\n\ndegrees_east\n\nvalid_max :\n\n180.0\n\nvalid_min :\n\n-180.0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n1.84 MiB\n611.91 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat64 numpy.ndarray\n\n\n\n\n\n\n\n\nData variables: (5)\n\n\n\n\n\natl06_quality_summary\n\n\n(delta_time)\n\n\nint8\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nqualityInformation\n\ndescription :\n\nThe ATL06_quality_summary parameter indicates the best-quality subset of all ATL06 data. A zero in this parameter implies that no data-quality tests have found a problem with the segment, a one implies that some potential problem has been found. Users who select only segments with zero values for this flag can be relatively certain of obtaining high-quality data, but will likely miss a significant fraction of usable data, particularly in cloudy, rough, or low-surface-reflectance conditions.\n\nflag_meanings :\n\nbest_quality potential_problem\n\nflag_values :\n\n[0 1]\n\nlong_name :\n\nATL06_Quality_Summary\n\nsource :\n\nsection 4.3\n\nunits :\n\n1\n\nvalid_max :\n\n1\n\nvalid_min :\n\n0\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n236.05 kiB\n76.49 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nint8 numpy.ndarray\n\n\n\n\n\n\n\n\n\nh_li\n\n\n(delta_time)\n\n\nfloat32\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nphysicalMeasurement\n\ndescription :\n\nStandard land-ice segment height determined by land ice algorithm, corrected for first-photon bias, representing the median- based height of the selected PEs\n\nlong_name :\n\nLand Ice height\n\nsource :\n\nsection 4.4\n\nunits :\n\nmeters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.92 MiB\n305.96 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nh_li_sigma\n\n\n(delta_time)\n\n\nfloat32\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nqualityInformation\n\ndescription :\n\nPropagated error due to sampling error and FPB correction from the land ice algorithm\n\nlong_name :\n\nExpected RMS segment misfit\n\nsource :\n\nsection 4.4\n\nunits :\n\nmeters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.92 MiB\n305.96 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nsegment_id\n\n\n(delta_time)\n\n\nfloat64\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nreferenceInformation\n\ndescription :\n\nSegment number, counting from the equator. Equal to the segment_id for the second of the two 20m ATL03 segments included in the 40m ATL06 segment\n\nlong_name :\n\nReference Point, m\n\nsource :\n\nsection 3.1.2.1\n\nunits :\n\n1\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n1.84 MiB\n611.91 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat64 numpy.ndarray\n\n\n\n\n\n\n\n\n\nsigma_geo_h\n\n\n(delta_time)\n\n\nfloat32\n\n\ndask.array<chunksize=(78325,), meta=np.ndarray>\n\n\n\n\ncontentType :\n\nqualityInformation\n\ndescription :\n\nTotal vertical geolocation error due to PPD and POD, including the effects of horizontal geolocation error on the segment vertical error.\n\nlong_name :\n\nVertical Geolocation Error\n\nsource :\n\nATBD Section 3.10\n\nunits :\n\nmeters\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nArray\nChunk\n\n\n\n\nBytes\n0.92 MiB\n305.96 kiB\n\n\nShape\n(241711,)\n(78325,)\n\n\nDask graph\n5 chunks in 11 graph layers\n\n\nData type\nfloat32 numpy.ndarray\n\n\n\n\n\n\n\n\n\nIndexes: (1)delta_timePandasIndexPandasIndex(DatetimeIndex(['2022-07-18 01:00:46.678760592',\n '2022-07-18 01:00:46.681322640',\n '2022-07-18 01:00:46.684008720',\n '2022-07-18 01:00:46.686753504',\n '2022-07-18 01:00:46.689526560',\n '2022-07-18 01:00:46.692315280',\n '2022-07-18 01:00:46.695049040',\n '2022-07-18 01:00:46.700724096',\n '2022-07-18 01:00:46.703545872',\n '2022-07-18 01:00:46.706366832',\n ...\n '2022-07-26 00:49:18.806914512',\n '2022-07-26 00:49:18.809737328',\n '2022-07-26 00:49:18.812559600',\n '2022-07-26 00:49:18.815380608',\n '2022-07-26 00:49:18.818200224',\n '2022-07-26 00:49:18.821015744',\n '2022-07-26 00:49:18.823827088',\n '2022-07-26 00:49:18.826637808',\n '2022-07-26 00:49:18.829449568',\n '2022-07-26 00:49:18.832263232'],\n dtype='datetime64[ns]', name='delta_time', length=241711, freq=None))Attributes: (2)Description :The land_ice_height group contains the primary set of derived ATL06 products. This includes geolocation, height, and standard error and quality measures for each segment. This group is sparse, meaning that parameters are provided only for pairs of segments for which at least one beam has a valid surface-height measurement.data_rate :Data within this group are sparse. Data values are provided only for those ICESat-2 20m segments where at least one beam has a valid land ice height measurement." }, { - "objectID": "external/appeears_csv_cloud_access.html#extract-the-direct-s3-links", - "href": "external/appeears_csv_cloud_access.html#extract-the-direct-s3-links", - "title": "How to work with AppEEARS Point Sample CSV outputs", - "section": "3. Extract the Direct S3 links", - "text": "3. Extract the Direct S3 links\nNow that we have our outputs ready, we can get the bundle information for the files included in the outputs. If you submitted your request using AppEEARS GUI, assign your sample’s task_id to the variable task_id below.\n\n#task_id = '<ADD task_id / request_id from AppEEARS GUI>'\n\nrequests.get is used toget the bundle information. Below, bundle information is printed. The bundle information includes s3_url in addition to the other information such as file_name, file_id, and file_type.\nEach output file can be downloaded using the file_id and AppEEARS API endpoint. AppEEARS outputs are also stored in an AWS bucket that can be accessed using S3_url.\n\nbundle = requests.get('{}bundle/{}'.format(appeears_API_endpoint,task_id), headers=head).json() # Call API and return bundle contents for the task_id as json\nbundle\n\n{'files': [{'sha256': '2d217eb9d967d849212bf6702a91e3fd615a7188cba72bf58e0b4d9849417865',\n 'file_id': '959e4c02-6ff1-4370-ba0d-67f5613eb3fa',\n 'file_name': 'EROS-MOD11A1-061-results.csv',\n 'file_size': 1168238,\n 'file_type': 'csv',\n 's3_url': 's3://appeears-output/9b2f9a77-da1a-41ac-ba47-baeb577ad99f/EROS-MOD11A1-061-results.csv'},\n {'sha256': '70dc8abc16c8368f6926f735f5f9057c27338a41f77165e648c600185acdff39',\n 'file_id': 'd198e3ac-93ce-4587-92df-f296c3d6b75b',\n 'file_name': 'EROS-granule-list.txt',\n 'file_size': 479520,\n 'file_type': 'txt',\n 's3_url': 's3://appeears-output/9b2f9a77-da1a-41ac-ba47-baeb577ad99f/EROS-granule-list.txt'},\n {'sha256': 'dd9359b792376d6939fac09b8be74ad5db8ca74e009aa386325f4d5ff0b1a004',\n 'file_id': '7c0280db-6764-4672-a58f-e8cc29229c32',\n 'file_name': 'EROS-request.json',\n 'file_size': 786,\n 'file_type': 'json',\n 's3_url': 's3://appeears-output/9b2f9a77-da1a-41ac-ba47-baeb577ad99f/EROS-request.json'},\n {'sha256': '3795b274a5aaceade145441ca09fe9a73925c64ef32d235923e66112007e142e',\n 'file_id': 'd1f49518-357a-45b7-b6ee-4b83411f1389',\n 'file_name': 'EROS-MOD11A1-061-metadata.xml',\n 'file_size': 16775,\n 'file_type': 'xml',\n 's3_url': 's3://appeears-output/9b2f9a77-da1a-41ac-ba47-baeb577ad99f/EROS-MOD11A1-061-metadata.xml'},\n {'sha256': '551b41b89b0a1130a00a6e35f4d6812f284253fe37f994cefeec18b92ce8967d',\n 'file_id': 'b926d235-70d0-4b63-aa5a-5abe32a032d4',\n 'file_name': 'README.md',\n 'file_size': 18943,\n 'file_type': 'txt',\n 's3_url': 's3://appeears-output/9b2f9a77-da1a-41ac-ba47-baeb577ad99f/README.md'}],\n 'created': '2023-03-13T16:05:31.183063',\n 'task_id': '9b2f9a77-da1a-41ac-ba47-baeb577ad99f',\n 'updated': '2023-03-13T16:41:31.639875',\n 'bundle_type': 'point'}\n\n\nBelow, the S3 Links to CSV output is filted.\n\nfile = [f for f in bundle['files'] if f['file_type'] == 'csv']\ncsv_urls = file[0]['s3_url']" + "objectID": "how-tos/access-data/access-cloud-python.html#end-user-license-agreement-eula", + "href": "how-tos/access-data/access-cloud-python.html#end-user-license-agreement-eula", + "title": "How do I access data stored in Earthdata Cloud in Python?", + "section": "End User License Agreement (EULA)", + "text": "End User License Agreement (EULA)\nSometimes, accessing data in NASA Earthdata Cloud requires an End-User License Agreement (EULA). If you cannot access a dataset, this may be your issue! See these instructions for how to authorize EULAs." }, { - "objectID": "external/appeears_csv_cloud_access.html#create-a-boto3-refreshable-session", - "href": "external/appeears_csv_cloud_access.html#create-a-boto3-refreshable-session", - "title": "How to work with AppEEARS Point Sample CSV outputs", - "section": "4. Create a boto3 Refreshable Session", - "text": "4. Create a boto3 Refreshable Session\nAppEEARS outputs are freely accessible from a cloud instance in us-west-2 region. In order to access our output files, a Boto3 session is needed. The Boto session will stores the required configurations for an easy integration between Python and AWS services. Below, get_boto3_refreshable_session stored in aws_session will access your Earthdata login credentidals store in .netrc file and generate S3 credential by making a call to AppEEARS S3 credential endpoint, and create a boto3 session. This session will be auto-renewed as needed to prevent timeouts errors related to S3 credentials.\n\nregion_name = 'us-west-2'\ns3_creds_endpoint = f\"{appeears_API_endpoint}/s3credentials\"\n# Boto3 Session required by the pandas package \nboto3_session = aws_session.get_boto3_refreshable_session(s3_creds_endpoint, region_name)\n\nINFO : Getting new temporary credentials from AppEEARS API https://appeears.earthdatacloud.nasa.gov/api//s3credentials...\n\n\nNext, an object for s3 client is created using boto3_session.\n\nboto3_client = boto3_session.client('s3', config=Config(max_pool_connections=20))" + "objectID": "how-tos/access-data/access-cloud-python.html#alternative-access-method-without-earthaccess", + "href": "how-tos/access-data/access-cloud-python.html#alternative-access-method-without-earthaccess", + "title": "How do I access data stored in Earthdata Cloud in Python?", + "section": "Alternative Access Method without earthaccess", + "text": "Alternative Access Method without earthaccess\nAn alternative approach to accessing data is outlined in some notebooks in the Earthdata Cloud Cookbook Appendix! The earthaccess package uses these methods for it’s back end. See this GitHub folder." }, { - "objectID": "external/appeears_csv_cloud_access.html#csv-file-in-region-direct-s3-access", - "href": "external/appeears_csv_cloud_access.html#csv-file-in-region-direct-s3-access", - "title": "How to work with AppEEARS Point Sample CSV outputs", - "section": "5. CSV File In-Region Direct S3 Access", - "text": "5. CSV File In-Region Direct S3 Access\nTo read the CSV file, an object is created for the bucket name along with the file name of csv file. AppEEARS stores the output csv file in a folder. Bucket and key can be extracted using S3 URL as below.\n\nbucket_name = csv_urls.split('/')[2]\nkey_name = csv_urls.split(bucket_name)[1].strip(\"/\")\nobj = boto3_client.get_object(Bucket = bucket_name, Key = key_name) \n\nNext, read_csv from pandas is used to read the body of created object.\n\noutput_csv = pandas.read_csv(obj['Body'])\noutput_csv\n\n\n\n\n\n\n\n\nID\nLatitude\nLongitude\nDate\nMODIS_Tile\nMOD11A1_061_Line_Y_1km\nMOD11A1_061_Sample_X_1km\nMOD11A1_061_LST_Day_1km\nMOD11A1_061_LST_Night_1km\nMOD11A1_061_QC_Day\n...\nMOD11A1_061_QC_Day_LST_Error_Flag_Description\nMOD11A1_061_QC_Night_bitmask\nMOD11A1_061_QC_Night_MODLAND\nMOD11A1_061_QC_Night_MODLAND_Description\nMOD11A1_061_QC_Night_Data_Quality_flag\nMOD11A1_061_QC_Night_Data_Quality_flag_Description\nMOD11A1_061_QC_Night_Emis_Error_flag\nMOD11A1_061_QC_Night_Emis_Error_flag_Description\nMOD11A1_061_QC_Night_LST_Error_Flag\nMOD11A1_061_QC_Night_LST_Error_Flag_Description\n\n\n\n\n0\nEROS\n43.736038\n-96.62139\n2015-01-01\nh11v04\n751.0\n22.0\n0.000000\n0.000000\n2.0\n...\nAverage LST error <= 1K\n0b00000010\n0b10\nLST not produced due to cloud effects\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n1\nEROS\n43.736038\n-96.62139\n2015-01-02\nh11v04\n751.0\n22.0\n268.940002\n0.000000\n65.0\n...\nAverage LST error <= 2K\n0b00000010\n0b10\nLST not produced due to cloud effects\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n2\nEROS\n43.736038\n-96.62139\n2015-01-03\nh11v04\n751.0\n22.0\n0.000000\n0.000000\n2.0\n...\nAverage LST error <= 1K\n0b00000010\n0b10\nLST not produced due to cloud effects\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n3\nEROS\n43.736038\n-96.62139\n2015-01-04\nh11v04\n751.0\n22.0\n256.259979\n0.000000\n65.0\n...\nAverage LST error <= 2K\n0b00000010\n0b10\nLST not produced due to cloud effects\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n4\nEROS\n43.736038\n-96.62139\n2015-01-05\nh11v04\n751.0\n22.0\n0.000000\n0.000000\n2.0\n...\nAverage LST error <= 1K\n0b00000010\n0b10\nLST not produced due to cloud effects\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n\n\n2955\nEROS\n43.736038\n-96.62139\n2023-02-25\nh11v04\n751.0\n22.0\n264.540008\n0.000000\n0.0\n...\nAverage LST error <= 1K\n0b00000010\n0b10\nLST not produced due to cloud effects\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n2956\nEROS\n43.736038\n-96.62139\n2023-02-26\nh11v04\n751.0\n22.0\n270.299988\n257.759979\n0.0\n...\nAverage LST error <= 1K\n0b00000000\n0b00\nLST produced, good quality, not necessary to e...\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n2957\nEROS\n43.736038\n-96.62139\n2023-02-27\nh11v04\n751.0\n22.0\n0.000000\n0.000000\n2.0\n...\nAverage LST error <= 1K\n0b00000010\n0b10\nLST not produced due to cloud effects\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n2958\nEROS\n43.736038\n-96.62139\n2023-02-28\nh11v04\n751.0\n22.0\n0.000000\n267.239990\n2.0\n...\nAverage LST error <= 1K\n0b00000000\n0b00\nLST produced, good quality, not necessary to e...\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n2959\nEROS\n43.736038\n-96.62139\n2023-03-01\nh11v04\n751.0\n22.0\n0.000000\n270.239990\n2.0\n...\nAverage LST error <= 1K\n0b01000001\n0b01\nLST produced, other quality, recommend examina...\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b01\nAverage LST error <= 2K\n\n\n\n\n2960 rows × 29 columns" + "objectID": "how-tos/find-data/earthdata_search.html", + "href": "how-tos/find-data/earthdata_search.html", + "title": "How do I find data using Earthdata Search?", + "section": "", + "text": "The original source for this document is https://nasa-openscapes.github.io/2021-Cloud-Workshop-AGU/tutorials/01_Earthdata_Search.html\nThis tutorial guides you through how to use Earthdata Search for NASA Earth observations search and discovery, and how to connect the search output (e.g. download or access links) to a programmatic workflow (locally or from within the cloud).\n\nStep 1. Go to Earthdata Search and Login\nGo to Earthdata Search https://search.earthdata.nasa.gov and use your Earthdata login credentials to log in. If you do not have an Earthdata account, please see the Workshop Prerequisites for guidance.\n\n\nStep 2. Search for dataset of interest\nUse the search box in the upper left to type key words. In this example we are interested in the ECCO dataset, hosted by the PO.DAAC. This dataset is available from the NASA Earthdata Cloud archive hosted in AWS cloud.\nClick on the “Available from AWS Cloud” filter option on the left. Here, 104 matching collections were found with the basic ECCO search.\n\n\n\nFigure caption: Search for ECCO data available in AWS cloud in Earthdata Search portal\n\n\nLet’s refine our search further. Let’s search for ECCO monthly SSH in the search box (which will produce 39 matching collections), and for the time period for year 2015. The latter can be done using the calendar icon on the left under the search box.\nScroll down the list of returned matches until we see the dataset of interest, in this case ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4).\nWe can click on the (i) icon for the dataset to read more details, including the dataset shortname (helpful for programmatic workflows) just below the dataset name; here ECCO_L4_SSH_05DEG_MONTHLY_V4R4.\n\n\n\nFigure caption: Refine search, set temporal bounds, get more information\n\n\n\n\nStep 3. Explore the dataset details, including Cloud Access information\nOnce we clicked the (i), scrolling down the info page for the dataset we will see Cloud Access information, such as:\n\nwhether the dataset is available in the cloud\nthe cloud Region (all NASA Earthdata Cloud data is/will be in us-west-2 region)\nthe S3 storage bucket and object prefix where this data is located\nlink that generates AWS S3 Credentials for in-cloud data access (we will cover this in the Direct Data Access Tutorials)\nlink to documentation describing the In-region Direct S3 Access to Buckets. Note: these will be unique depending on the DAAC where the data is archived. (We will show examples of direct in-region access in Tutorial 3.)\n\n\n\n\nFigure caption: Cloud access info in EDS\n\n\n\n\n\nFigure caption: Documentation describing the In-region Direct S3 Access to Buckets\n\n\nPro Tip: Clicking on “For Developers” to exapnd will provide programmatic endpoints such as those for the CMR API, and more. CMR API and CMR STAC API tutorials can be found on the 2021 Cloud Hackathon website.\nFor now, let’s say we are intersted in getting download link(s) or access link(s) for specific data files (granules) within this collection.\nAt the top of the dataset info section, click on Search Results, which will take us back to the list of datasets matching our search parameters. Clicking on the dataset (here again it’s the same ECCO Sea Surface Height - Monthly Mean 0.5 Degree (Version 4 Release 4)) we now see a list of files (granules) that are part of the dataset (collection).\n\n\nStep 4. Customize the download or data access\nClick on the green + symbol to add a few files to our project. Here we added the first 3 listed for 2015. Then click on the green button towards the bottom that says “Download”. This will take us to another page with options to customize our download or access link(s).\n\n\n\nFigure caption: Select granules and click download\n\n\n\n4.a. Entire file content\nLet’s stay we are interested in the entire file content, so we select the “Direct Download” option (as opposed to other options to subset or transform the data):\n\n\n\nFigure caption: Customize your download or access\n\n\nClicking the green Download Data button again, will take us to the final page for instructions to download and links for data access in the cloud. You should see three tabs: Download Files, AWS S3 Access, Download Script:\n \nThe Download Files tab provides the https:// links for downloading the files locally. E.g.: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-09_ECCO_V4r4_latlon_0p50deg.nc\nThe AWS S3 Access tab provides the S3:// links, which is what we would use to access the data directly in-region (us-west-2) within the AWS cloud (an example will be shown in Tutorial 3). E.g.: s3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-09_ECCO_V4r4_latlon_0p50deg.nc where s3 indicates data is stored in AWS S3 storage, podaac-ops-cumulus-protected is the bucket, and ECCO_L4_SSH_05DEG_MONTHLY_V4R4 is the object prefix (the latter two are also listed in the dataset collection information under Cloud Access (step 3 above)).\nTip: Another quicker way to find the bucket and object prefix is from the list of data files the search returns. Next to the + green button is a grey donwload symbol. Click on that to see the Download Files https:// links or on the AWS S3 Access to get the direct S3:// access links, which contain the bucket and object prefix where data is stored.\n\n\n4.b. Subset or transform before download or access\nDAAC tools and services are also being migrated or developed in the cloud, next to that data. These include the Harmony API and OPeNDAP in the cloud, as a few examples.\nWe can leverage these cloud-based services on cloud-archived data to reduce or transform the data (depending on need) before getting the access links regardless of whether we prefer to download the data and work on a local machine or whether we want to access the data in the cloud (from a cloud workspace). These can be useful data reduction services that support a faster time to science.\nHarmony\nHarmony allows you to seamlessly analyze Earth observation data from different NASA data centers. These services (API endpoints) provide data reduction (e.g. subsetting) and transfromation services (e.g. convert netCDF data to Zarr cloud optimized format).\n\n\n\nFigure caption: Leverage Harmony cloud-based data transformation services\n\n\nWhen you click the final green Download button, the links provided are to data that had been transformed based on our selections on the previous screen (here chosing to use the Harmony service to reformat the data to Zarr). These data are staged for us in an S3 bucket in AWS, and we can use the s3:// links to access those specific data. This service also provides STAC access links. This particular example is applicable if your workflow is in the AWS us-west-2 region.\n\n\n\nFigure caption: Harmony-staged data in S3\n\n\n\n\n\nStep 5. Integrate file links into programmatic workflow, locally or in the AWS cloud.\nIn tutorial 3 Direct Data Access, we will work programmatically in the cloud to access datasets of interest, to get us set up for further scientific analysis of choice. There are several ways to do this. One way to connect the search part of the workflow we just did in Earthdata Search to our next steps working in the cloud is to simply copy/paste the s3:// links provides in Step 4 above into a JupyterHub notebook or script in our cloud workspace, and continue the data analysis from there.\nOne could also copy/paste the s3:// links and save them in a text file, then open and read the text file in the notebook or script in the JupyterHub in the cloud.\nTutorial 3 will pick up from here and cover these next steps in more detail." }, { - "objectID": "external/appeears_csv_cloud_access.html#quality-filtering", - "href": "external/appeears_csv_cloud_access.html#quality-filtering", - "title": "How to work with AppEEARS Point Sample CSV outputs", - "section": "6. Quality Filtering", - "text": "6. Quality Filtering\nTo get the good quality data, the quality unique values are assigned to a variable. Good quality data have a decoded quality flag as LST produced, good quality, not necessary to examine more detailed QA. The LST values equal to zero, the layer fill_value, are also filtered.\n\nquality_val = numpy.array(output_csv['MOD11A1_061_QC_Night_MODLAND_Description'])\nquality_unique = numpy.unique(quality_val)\nquality_unique\n\narray(['LST not produced due to cloud effects',\n 'LST produced, good quality, not necessary to examine more detailed QA',\n 'LST produced, other quality, recommend examination of more detailed QA'],\n dtype=object)\n\n\n\ngood_qual_flag = 'LST produced, good quality, not necessary to examine more detailed QA'\ngoodQual = output_csv[output_csv['MOD11A1_061_QC_Night_MODLAND_Description'] == good_qual_flag]\ngoodQual = goodQual[goodQual['MOD11A1_061_LST_Day_1km'] != 0]" + "objectID": "how-tos/find-data/programmatic.html", + "href": "how-tos/find-data/programmatic.html", + "title": "How do I find data using code?", + "section": "", + "text": "Here are our recommended approaches for finding data with code, from the command line or a notebook.\n\nPythonRMatlabCommand Line\n\n\nIn Python we can use the earthaccess library (renamed, previously earthdata)\nTo install the package we’ll run this code from the command line. Note: you can run shell code directly from a Jupyter Notebook cell by adding a !, so it would be !conda install.\n## In the command line\n## Install earthaccess\nconda install -c conda-forge earthaccess\nThis example searches for data from the Land Processes DAAC with a spatial bounding box and temporal range.\n## In Python\n## Import packages\nfrom earthaccess import DataGranules, DataCollections\nfrom pprint import pprint \n\n\n## We'll get 4 collections that match with our keyword of interest\ncollections = DataCollections().keyword(\"REFLECTANCE\").cloud_hosted(True).get(4)\n\n## Let's print 2 collections\nfor collection in collections[0:2]:\n print(pprint(collection.summary()) , collection.abstract(), \"\\n\")\n \n## Search for files from the second dataset result over a small plot in Nebraska, USA for two weeks in September 2022\ngranules = DataGranules().concept_id(\"C2021957657-LPCLOUD\").temporal(\"2022-09-10\",\"2022-09-24\").bounding_box(-101.67271,41.04754,-101.65344,41.06213)\nprint(len(granules))\ngranules\n\n\nTo find data in R, we’ll also use the earthaccess python package - we can do so from R using the reticulate package (cheatsheet). Note below that we import the python library as an R object we name earthaccess, as well as the earthaccess$ syntax for accessing functions from the earthaccess library. The granules object has a list of JSON dictionaries with some extra dictionaries.\n## In R\n## load R libraries\nlibrary(tidyverse) # install.packages(\"tidyverse\") \nlibrary(reticulate) # install.packages(\"reticulate\")\n\n## load python library\nearthaccess <- reticulate::import(\"earthaccess\") \n\n## use earthaccess to access data # https://nsidc.github.io/earthaccess/tutorials/search-granules/\ngranules <- earthaccess$search_data(\n doi = \"10.5067/SLREF-CDRV3\",\n temporal = reticulate::tuple(\"2017-01\", \"2017-02\") # with an earthaccess update, this can be simply c() or list()\n)\n\n## Granules found: 72\n\n## exploring\ngranules # this is the result of the get request. \n\nclass(granules) # \"list\"\n## granules <- reticulate::py_to_r(granules) # Object to convert is not a Python object\n\n\nMatlab code coming soon!\n## In Matlab\n## Coming soon!\n\n\nWith wget and curl:\n## In the command line\n## Coming soon!" }, { - "objectID": "external/appeears_csv_cloud_access.html#explore-the-lst-time-series", - "href": "external/appeears_csv_cloud_access.html#explore-the-lst-time-series", - "title": "How to work with AppEEARS Point Sample CSV outputs", - "section": "6. Explore the LST Time Series", - "text": "6. Explore the LST Time Series\nThe matplotlib can be used to create visualizations for a Pandas DataFrames. pyplot is used below to visualize non-Nan day and night LST observations flagged with the good quality.\n\nimport matplotlib.pyplot as plt\n\n\nfig, ax1 = plt.subplots(figsize=(15, 8))\n\nax1.plot(goodQual['Date'], goodQual['MOD11A1_061_LST_Day_1km'] - 273.15, color=\"#6495ED\", lw=2)\nax1.plot(goodQual['Date'], goodQual['MOD11A1_061_LST_Night_1km'] - 273.15, color=\"#CD3333\", lw=2)\n\nax1.set_xlabel(\"Date\")\nax1.set_ylabel(\"Land Surface Temperature (Celsius °)\", fontsize=14)\n\nax1.set_xticks (goodQual['Date'][::14]) # print pone label for every 14 obs \n\nfig.suptitle(\"Day and Night LST\", fontsize=20)\nfig.autofmt_xdate()\n\nINFO : Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.\nINFO : Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.\nINFO : Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.\nINFO : Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.\nINFO : Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting." + "objectID": "how-tos/find-data/programmatic.html#introduction", + "href": "how-tos/find-data/programmatic.html#introduction", + "title": "How do I find data using code?", + "section": "", + "text": "Here are our recommended approaches for finding data with code, from the command line or a notebook.\n\nPythonRMatlabCommand Line\n\n\nIn Python we can use the earthaccess library (renamed, previously earthdata)\nTo install the package we’ll run this code from the command line. Note: you can run shell code directly from a Jupyter Notebook cell by adding a !, so it would be !conda install.\n## In the command line\n## Install earthaccess\nconda install -c conda-forge earthaccess\nThis example searches for data from the Land Processes DAAC with a spatial bounding box and temporal range.\n## In Python\n## Import packages\nfrom earthaccess import DataGranules, DataCollections\nfrom pprint import pprint \n\n\n## We'll get 4 collections that match with our keyword of interest\ncollections = DataCollections().keyword(\"REFLECTANCE\").cloud_hosted(True).get(4)\n\n## Let's print 2 collections\nfor collection in collections[0:2]:\n print(pprint(collection.summary()) , collection.abstract(), \"\\n\")\n \n## Search for files from the second dataset result over a small plot in Nebraska, USA for two weeks in September 2022\ngranules = DataGranules().concept_id(\"C2021957657-LPCLOUD\").temporal(\"2022-09-10\",\"2022-09-24\").bounding_box(-101.67271,41.04754,-101.65344,41.06213)\nprint(len(granules))\ngranules\n\n\nTo find data in R, we’ll also use the earthaccess python package - we can do so from R using the reticulate package (cheatsheet). Note below that we import the python library as an R object we name earthaccess, as well as the earthaccess$ syntax for accessing functions from the earthaccess library. The granules object has a list of JSON dictionaries with some extra dictionaries.\n## In R\n## load R libraries\nlibrary(tidyverse) # install.packages(\"tidyverse\") \nlibrary(reticulate) # install.packages(\"reticulate\")\n\n## load python library\nearthaccess <- reticulate::import(\"earthaccess\") \n\n## use earthaccess to access data # https://nsidc.github.io/earthaccess/tutorials/search-granules/\ngranules <- earthaccess$search_data(\n doi = \"10.5067/SLREF-CDRV3\",\n temporal = reticulate::tuple(\"2017-01\", \"2017-02\") # with an earthaccess update, this can be simply c() or list()\n)\n\n## Granules found: 72\n\n## exploring\ngranules # this is the result of the get request. \n\nclass(granules) # \"list\"\n## granules <- reticulate::py_to_r(granules) # Object to convert is not a Python object\n\n\nMatlab code coming soon!\n## In Matlab\n## Coming soon!\n\n\nWith wget and curl:\n## In the command line\n## Coming soon!" }, { - "objectID": "external/appeears_csv_cloud_access.html#contact-info", - "href": "external/appeears_csv_cloud_access.html#contact-info", - "title": "How to work with AppEEARS Point Sample CSV outputs", - "section": "Contact Info:", - "text": "Contact Info:\nEmail: LPDAAC@usgs.gov\nVoice: +1-866-573-3222\nOrganization: Land Processes Distributed Active Archive Center (LP DAAC)¹\nWebsite: https://lpdaac.usgs.gov/\nDate last modified: 05-06-2023\n¹Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I." + "objectID": "how-tos/subset.html", + "href": "how-tos/subset.html", + "title": "How do I subset data granules?", + "section": "", + "text": "PythonRMATLABCommand Line\n\n\nInstall the harmony-py package:\n# Install harmony-py\npip install -U harmony-py\nImport packages:\nimport datetime as dt\n\nfrom harmony import BBox, Client, Collection, Request, LinkType\n\nimport s3fs\nimport xarray as xr\n\n\nWe will authenticate the following Harmony request using a netrc file. See the appendix for more information on Earthdata Login and netrc setup. This basic line below to create a Harmony Client assumes that we have a .netrc available.\nharmony_client = Client()\n\n\n\nWe are interested in the GHRSST Level 4 MUR Global Foundation Sea Surface Temperature Analysis dataset. We are subsetting over the Pacific Ocean to the west of Mexico during 1:00 - 2:00 on 10 March 2021. The dataset is organized into daily files, so while we are specifying a single hour in our request, this will return that full day’s worth of data.\ndataset_short_name = 'MUR-JPL-L4-GLOB-v4.1'\n\nrequest = Request(\n collection=Collection(id=dataset_short_name),\n spatial=BBox(-125.469,15.820,-99.453,35.859),\n temporal={\n 'start': dt.datetime(2021, 3, 10, 1),\n 'stop': dt.datetime(2021, 3, 10, 2)\n }\n)\n\njob_id = harmony_client.submit(request)\n\nharmony_client.wait_for_processing(job_id)\n\n\n\nHarmony data outputs can be accessed within the cloud using the s3 URLs and AWS credentials provided in the Harmony job response. Using aws_credentials we can retrieve the credentials needed to access the Harmony s3 staging bucket and its contents. We then use the AWS s3fs package to create a file system that can then be read by xarray.\nresults = harmony_client.result_urls(job_id, link_type=LinkType.s3)\nurls = list(results)\nurl = urls[0]\n\ncreds = harmony_client.aws_credentials()\n\ns3_fs = s3fs.S3FileSystem(\n key=creds['aws_access_key_id'],\n secret=creds['aws_secret_access_key'],\n token=creds['aws_session_token'],\n client_kwargs={'region_name':'us-west-2'},\n)\n\nf = s3_fs.open(url, mode='rb')\nds = xr.open_dataset(f)\nds\n\n\n\nUse the xarray built in plotting function to create a simple plot along the x and y dimensions of the dataset:\nds.analysed_sst.plot() ;\n\n\n\nR code coming soon!\n# Coming soon!\n\n\nThis example assumes that you have experimented with and understood the tutorial MATLAB Access Single NASA EarthData L2 NetCDF presented in this Cookbook.\nTo subset a dataset, either read it over its full extent and subset it using MATLAB indexing (a posteriori, client-side), or use the start, count, and stride parameters of function h5read to read only specific regions/samples (hyperslabs) of it (server side). The code snippets that follow present both approaches, for accessing a dataset of sea surface temperature.\nFollow the authentication procedure presented in the tutorial:\naddpath(\"<PATH_TO_FUNCTION_LOADAWSCREDENTIALSENPOINT>\") ;\ndaacCredentialsEndpoint = \"https://archive.podaac.earthdata.nasa.gov/s3credentials\";\nloadAWSCredentials(daacCredentialsEndpoint);\nDefine relevant file and dataset:\nfilePath = \"s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20100619062008-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\";\ndatasetName = \"sea_surface_temperature_4um\";\ndatasetPath = \"/\" + datasetName;\nRead full dataset, longitudes, and latitudes. Replace fill values by NaN (so they are not displayed by MALTAB), and build contour plot:\nfill_value = h5readatt(filePath,datasetPath,\"_FillValue\");\nname = h5readatt(filePath,datasetPath,\"long_name\");\ndata = h5read(filePath,datasetPath);\ndata(data == fill_value) = NaN;\nlat = h5read(filePath,\"/lat\");\nlon = h5read(filePath,\"/lon\");\ncontour(lon,lat,data);\ntitle(name);\nSubset/sub-sample full dataset with a step of 10 (a posteriori) using MATLAB indexing:\nstep = 10;\ndataSize = size(data);\nrowIx = 1 : step : dataSize(1);\ncolIx = 1 : step : dataSize(2);\ncontour(lon(rowIx,colIx),lat(rowIx,colIx),data(rowIx,colIx));\ntitle(name + \" (sub-sampled through MATLAB indexing)\");\nUse the start, count, and stride parameters of function h5read to read only specific regions/samples (hyperslabs) of it (server side). These parameters must be defined as vectors whose size matches the number of dimensions of the dataset. Here, we want to read data starting at index 1, with a stride/step of 10, and cover the full extent of the dataset (count = Inf). Based on our experiments above, we assume that the dataset is a 2D array:\ndataStride10 = h5read(filePath,datasetPath,[1,1],[Inf,Inf],[step,step]);\nRunning this leads to the following error:\nError using h5readc\nThe START, COUNT, and STRIDE parameters must be double precision, positive, and have length equal to the rank of the dataset.\nThis error message is too succint to be helpful. Try to get more information about the dataset using the h5disp, which is cloud-enabled unlike ncdisp:\nh5disp(filePath,datasetPath)\n\nOUTPUT:\nHDF5 20100619062008-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc \nDataset 'sea_surface_temperature_4um' \n Size: 1354x2030x1 <--- Shows a 3rd (singleton) dimension\n MaxSize: 1354x2030x1 \n Datatype: H5T_STD_I16LE (int16)\n ChunkSize: 677x1015x1\n Filters: deflate(5)\n FillValue: -32767\n Attributes:\n 'long_name': 'sea surface temperature'\n 'units': 'kelvin'\n '_FillValue': -32767 \n 'valid_min': -1000 \n 'valid_max': 10000 \n 'comment': 'sea surface temperature from mid-IR (4 um) channels; non L2P core field'\n 'scale_factor': 0.005000 \n 'add_offset': 273.149994 \n 'coordinates': 'lon lat'\n 'coverage_content_type': 'physicalMeasurement'\n 'DIMENSION_LIST': H5T_VLEN\nThis output shows a 3rd (singleton) dimension, but it still does not explain what it is.\nOne workaround is to display more of the content of the file through h5disp and/or h5info and try to get relevant information about dimensions. The following shows a simpler approach, however: create a local copy of the file and analyze it using ncdisp. It is possible because MATLAB IO functions, and in particular copyfile, are cloud-enabled:\ncopyfile(filePath,\"test.nc\");\nncdisp(\"test.nc\",datasetName);\n\nOUTPUT:\nSource:\n .../test.nc\nFormat:\n netcdf4\nDimensions:\n ni = 1354\n nj = 2030\n time = 1\nVariables:\n sea_surface_temperature_4um\n Size: 1354x2030x1\n Dimensions: ni,nj,time\n Datatype: int16\n Attributes:\n ...\nThis makes it clear that there is a 3rd dimension that is time, with a size of 1.\nSubset the dataset using h5read and vectors of 3 elements that define start, count, and stride for the 3 dimensions:\ndataStride10 = h5read(filePath,datasetPath,[1,1,1],[Inf,Inf,Inf],[step,step,1]);\ndataStride10(dataStride10 == fill_value) = NaN;\nlatStride10 = h5read(filePath,\"/lat\",[1,1],[Inf,Inf],[step,step]);\nlonStride10 = h5read(filePath,\"/lon\",[1,1],[Inf,Inf],[step,step]);\ncontour(lonStride10,latStride10,dataStride10);\ntitle(name+ \" (sub-sampled through H5READ stride parameter)\");\nWe can target a specific region using the same two approaches. The following shows how to create an array of logicals that “flags” a region of interest, how to identify relevant rows and columns of the data, latitudes, and longitudes arrays, and how to use logical indexing for plotting the data:\nlatBounds = [31,33.5];\nlonBounds = [-64,-57.5];\nisRegion = lat>=latBounds(1) & lat<=latBounds(2) & lon>=lonBounds(1) & lon<=lonBounds(2);\nisRow = any(isRegion,2);\nisCol = any(isRegion,1);\ncontour(lon(isRow,isCol),lat(isRow,isCol),data(isRow,isCol));\nxlim(lonBounds);\nylim(latBounds);\ntitle(name+ \" (selected region)\");\n\n\nWith wget and curl:\n# Coming soon!" }, { - "objectID": "external/cof-zarr-reformat.html", - "href": "external/cof-zarr-reformat.html", - "title": "COF Zarr Access via Reformat", + "objectID": "how-tos/subset.html#how-do-i-subset-a-data-granule-using-harmony", + "href": "how-tos/subset.html#how-do-i-subset-a-data-granule-using-harmony", + "title": "How do I subset data granules?", "section": "", - "text": "imported on: 2023-05-30\nThis notebook is from NASA’s PO.DAAC, Access ECCO data via Harmony and the Zarr reformatter service example" + "text": "PythonRMATLABCommand Line\n\n\nInstall the harmony-py package:\n# Install harmony-py\npip install -U harmony-py\nImport packages:\nimport datetime as dt\n\nfrom harmony import BBox, Client, Collection, Request, LinkType\n\nimport s3fs\nimport xarray as xr\n\n\nWe will authenticate the following Harmony request using a netrc file. See the appendix for more information on Earthdata Login and netrc setup. This basic line below to create a Harmony Client assumes that we have a .netrc available.\nharmony_client = Client()\n\n\n\nWe are interested in the GHRSST Level 4 MUR Global Foundation Sea Surface Temperature Analysis dataset. We are subsetting over the Pacific Ocean to the west of Mexico during 1:00 - 2:00 on 10 March 2021. The dataset is organized into daily files, so while we are specifying a single hour in our request, this will return that full day’s worth of data.\ndataset_short_name = 'MUR-JPL-L4-GLOB-v4.1'\n\nrequest = Request(\n collection=Collection(id=dataset_short_name),\n spatial=BBox(-125.469,15.820,-99.453,35.859),\n temporal={\n 'start': dt.datetime(2021, 3, 10, 1),\n 'stop': dt.datetime(2021, 3, 10, 2)\n }\n)\n\njob_id = harmony_client.submit(request)\n\nharmony_client.wait_for_processing(job_id)\n\n\n\nHarmony data outputs can be accessed within the cloud using the s3 URLs and AWS credentials provided in the Harmony job response. Using aws_credentials we can retrieve the credentials needed to access the Harmony s3 staging bucket and its contents. We then use the AWS s3fs package to create a file system that can then be read by xarray.\nresults = harmony_client.result_urls(job_id, link_type=LinkType.s3)\nurls = list(results)\nurl = urls[0]\n\ncreds = harmony_client.aws_credentials()\n\ns3_fs = s3fs.S3FileSystem(\n key=creds['aws_access_key_id'],\n secret=creds['aws_secret_access_key'],\n token=creds['aws_session_token'],\n client_kwargs={'region_name':'us-west-2'},\n)\n\nf = s3_fs.open(url, mode='rb')\nds = xr.open_dataset(f)\nds\n\n\n\nUse the xarray built in plotting function to create a simple plot along the x and y dimensions of the dataset:\nds.analysed_sst.plot() ;\n\n\n\nR code coming soon!\n# Coming soon!\n\n\nThis example assumes that you have experimented with and understood the tutorial MATLAB Access Single NASA EarthData L2 NetCDF presented in this Cookbook.\nTo subset a dataset, either read it over its full extent and subset it using MATLAB indexing (a posteriori, client-side), or use the start, count, and stride parameters of function h5read to read only specific regions/samples (hyperslabs) of it (server side). The code snippets that follow present both approaches, for accessing a dataset of sea surface temperature.\nFollow the authentication procedure presented in the tutorial:\naddpath(\"<PATH_TO_FUNCTION_LOADAWSCREDENTIALSENPOINT>\") ;\ndaacCredentialsEndpoint = \"https://archive.podaac.earthdata.nasa.gov/s3credentials\";\nloadAWSCredentials(daacCredentialsEndpoint);\nDefine relevant file and dataset:\nfilePath = \"s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20100619062008-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\";\ndatasetName = \"sea_surface_temperature_4um\";\ndatasetPath = \"/\" + datasetName;\nRead full dataset, longitudes, and latitudes. Replace fill values by NaN (so they are not displayed by MALTAB), and build contour plot:\nfill_value = h5readatt(filePath,datasetPath,\"_FillValue\");\nname = h5readatt(filePath,datasetPath,\"long_name\");\ndata = h5read(filePath,datasetPath);\ndata(data == fill_value) = NaN;\nlat = h5read(filePath,\"/lat\");\nlon = h5read(filePath,\"/lon\");\ncontour(lon,lat,data);\ntitle(name);\nSubset/sub-sample full dataset with a step of 10 (a posteriori) using MATLAB indexing:\nstep = 10;\ndataSize = size(data);\nrowIx = 1 : step : dataSize(1);\ncolIx = 1 : step : dataSize(2);\ncontour(lon(rowIx,colIx),lat(rowIx,colIx),data(rowIx,colIx));\ntitle(name + \" (sub-sampled through MATLAB indexing)\");\nUse the start, count, and stride parameters of function h5read to read only specific regions/samples (hyperslabs) of it (server side). These parameters must be defined as vectors whose size matches the number of dimensions of the dataset. Here, we want to read data starting at index 1, with a stride/step of 10, and cover the full extent of the dataset (count = Inf). Based on our experiments above, we assume that the dataset is a 2D array:\ndataStride10 = h5read(filePath,datasetPath,[1,1],[Inf,Inf],[step,step]);\nRunning this leads to the following error:\nError using h5readc\nThe START, COUNT, and STRIDE parameters must be double precision, positive, and have length equal to the rank of the dataset.\nThis error message is too succint to be helpful. Try to get more information about the dataset using the h5disp, which is cloud-enabled unlike ncdisp:\nh5disp(filePath,datasetPath)\n\nOUTPUT:\nHDF5 20100619062008-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc \nDataset 'sea_surface_temperature_4um' \n Size: 1354x2030x1 <--- Shows a 3rd (singleton) dimension\n MaxSize: 1354x2030x1 \n Datatype: H5T_STD_I16LE (int16)\n ChunkSize: 677x1015x1\n Filters: deflate(5)\n FillValue: -32767\n Attributes:\n 'long_name': 'sea surface temperature'\n 'units': 'kelvin'\n '_FillValue': -32767 \n 'valid_min': -1000 \n 'valid_max': 10000 \n 'comment': 'sea surface temperature from mid-IR (4 um) channels; non L2P core field'\n 'scale_factor': 0.005000 \n 'add_offset': 273.149994 \n 'coordinates': 'lon lat'\n 'coverage_content_type': 'physicalMeasurement'\n 'DIMENSION_LIST': H5T_VLEN\nThis output shows a 3rd (singleton) dimension, but it still does not explain what it is.\nOne workaround is to display more of the content of the file through h5disp and/or h5info and try to get relevant information about dimensions. The following shows a simpler approach, however: create a local copy of the file and analyze it using ncdisp. It is possible because MATLAB IO functions, and in particular copyfile, are cloud-enabled:\ncopyfile(filePath,\"test.nc\");\nncdisp(\"test.nc\",datasetName);\n\nOUTPUT:\nSource:\n .../test.nc\nFormat:\n netcdf4\nDimensions:\n ni = 1354\n nj = 2030\n time = 1\nVariables:\n sea_surface_temperature_4um\n Size: 1354x2030x1\n Dimensions: ni,nj,time\n Datatype: int16\n Attributes:\n ...\nThis makes it clear that there is a 3rd dimension that is time, with a size of 1.\nSubset the dataset using h5read and vectors of 3 elements that define start, count, and stride for the 3 dimensions:\ndataStride10 = h5read(filePath,datasetPath,[1,1,1],[Inf,Inf,Inf],[step,step,1]);\ndataStride10(dataStride10 == fill_value) = NaN;\nlatStride10 = h5read(filePath,\"/lat\",[1,1],[Inf,Inf],[step,step]);\nlonStride10 = h5read(filePath,\"/lon\",[1,1],[Inf,Inf],[step,step]);\ncontour(lonStride10,latStride10,dataStride10);\ntitle(name+ \" (sub-sampled through H5READ stride parameter)\");\nWe can target a specific region using the same two approaches. The following shows how to create an array of logicals that “flags” a region of interest, how to identify relevant rows and columns of the data, latitudes, and longitudes arrays, and how to use logical indexing for plotting the data:\nlatBounds = [31,33.5];\nlonBounds = [-64,-57.5];\nisRegion = lat>=latBounds(1) & lat<=latBounds(2) & lon>=lonBounds(1) & lon<=lonBounds(2);\nisRow = any(isRegion,2);\nisCol = any(isRegion,1);\ncontour(lon(isRow,isCol),lat(isRow,isCol),data(isRow,isCol));\nxlim(lonBounds);\nylim(latBounds);\ntitle(name+ \" (selected region)\");\n\n\nWith wget and curl:\n# Coming soon!" }, { - "objectID": "external/cof-zarr-reformat.html#getting-started", - "href": "external/cof-zarr-reformat.html#getting-started", - "title": "COF Zarr Access via Reformat", - "section": "Getting Started", - "text": "Getting Started\nWe will access monthly ocean bottom pressure (OBP) data from ECCO V4r4 (10.5067/ECG5M-OBP44), which are provided as a monthly time series on a 0.5-degree latitude/longitude grid.\nThe data are archived in netCDF format. However, this notebook demonstration will request conversion to Zarr format for files covering the period between 2010 and 2018. Upon receiving our request, Harmony’s backend will convert the files and stage them in S3 for native access in AWS (us-west-2 region, specifically). We will access the new Zarr datasets as an aggregated dataset using xarray, and leverage the S3 native protocols for direct access to the data in an efficient manner.\n\n\nRequirements\n\nAWS\nThis notebook should be running in an EC2 instance in AWS region us-west-2, as previously mentioned. We recommend using an EC2 with at least 8GB of memory available.\nThe notebook was developed and tested using a t2.large instance (2 cpus; 8GB memory).\n\n\nPython 3\nMost of these imports are from the Python standard library. However, you will need to install these packages into your Python 3 environment if you have not already done so:\n\ns3fs\nrequests\npandas\nxarray\nmatplotlib\n\n\n\n\nRequirements\n\nimport matplotlib.pyplot as plt\nimport xarray as xr\nimport pandas as pd\nimport numpy as np\nimport requests\nimport json\nimport time\nimport s3fs\n\nShortName = \"ECCO_L4_OBP_05DEG_MONTHLY_V4R4\"\n\n\n\nStudy period\nSet some “master” inputs to define the time and place contexts for our case studies in the ipynb. This example will be requesting time subsets and receiving global data back from the Harmony API.\n\nstart_date = \"2010-01-01\"\nend_date = \"2018-12-31\"\n\n\n\nData Access\nSome features in the Harmony API require us to identify the target dataset/collection by its concept-id (which uniquely idenfifies it among the other datasets in the Common Metadata Repository). Support for selection by the dataset ShortName will be added in a future release.\n\nCommon Metadata Repository (CMR)\nFor now, we will need to get the concept-id that corresponds to our dataset by accessing its metadata from the CMR. Read more about the CMR at: https://cmr.earthdata.nasa.gov/\nRequest the UMM Collection metadata (i.e. metadata about the dataset) from the CMR and select the concept-id as a new variable ccid.\n\nresponse = requests.get(\n url='https://cmr.earthdata.nasa.gov/search/collections.umm_json', \n params={'provider': \"POCLOUD\",\n 'ShortName': ShortName,\n 'page_size': 1}\n)\n\nummc = response.json()['items'][0]\n\nccid = ummc['meta']['concept-id']\n\nccid\n\n'C1990404791-POCLOUD'\n\n\n\n\nHarmony API\nAnd get the Harmony API endpoint and zarr parameter like we did for SMAP before:\n\nbase = f\"https://harmony.earthdata.nasa.gov/{ccid}\"\nhreq = f\"{base}/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset\"\nrurl = f\"{hreq}?format=application/x-zarr\"\n\nprint(rurl)\n\nhttps://harmony.earthdata.nasa.gov/C1990404791-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?format=application/x-zarr\n\n\nECCO monthly collections have 312 granules in V4r4 (you can confirm with the granule listing from CMR Search API) so we can get the entire time series for 2010 to 2018 with one request to the Harmony API.\nFormat a string of query parameters to limit the processing to the desired time period. Then, append the string of time subset parameters to the variable rurl.\n\nsubs = '&'.join([f'subset=time(\"{start_date}T00:00:00.000Z\":\"{end_date}T23:59:59.999Z\")'])\n\nrurl = f\"{rurl}&{subs}\"\n\nprint(rurl)\n\nhttps://harmony.earthdata.nasa.gov/C1990404791-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?format=application/x-zarr&subset=time(\"2010-01-01T00:00:00.000Z\":\"2018-12-31T23:59:59.999Z\")\n\n\nSubmit the request and monitor the processing status in a while loop, breaking it on completion of the request job:\n\nresponse = requests.get(url=rurl).json()\n\n# Monitor status in a while loop. Wait 10 seconds for each check.\nwait = 10\nwhile True:\n response = requests.get(url=response['links'][0]['href']).json()\n if response['status']!='running':\n break\n print(f\"Job in progress ({response['progress']}%)\")\n time.sleep(wait)\n\nprint(\"DONE!\")\n\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nDONE!\n\n\nAccess the staged cloud datasets over native AWS interfaces\nCheck the message field in the response for clues about how to proceed:\n\nprint(response['message'])\n\nThe job has completed successfully. Contains results in AWS S3. Access from AWS us-west-2 with keys from https://harmony.earthdata.nasa.gov/cloud-access.sh\n\n\nThe third item in the list of links contains the shell script from the job status message printed above. Let’s download the same information in JSON format. It should be the fourth item; check to be sure:\n\nlen(response['links'])\n\n102\n\n\nSelect the url and download the json, then load to Python dictionary and print the keys:\n\nwith requests.get(response['links'][3]['href']) as r:\n creds = r.json()\n\nprint(creds.keys())\n\ndict_keys(['AccessKeyId', 'SecretAccessKey', 'SessionToken', 'Expiration'])\n\n\nCheck the expiration timestamp for the temporary credentials:\n\ncreds['Expiration']\n\n'2021-06-11T02:36:29.000Z'\n\n\nOpen zarr datasets with s3fs and xarray\nGet the s3 output directory and list of zarr datasets from the list of links. The s3 directory should be the fifth item; the urls are from item six onward:\n\ns3_dir = response['links'][4]['href']\n\nprint(s3_dir)\n\ns3://harmony-prod-staging/public/harmony/netcdf-to-zarr/2295236b-8086-4543-9482-f524a9f2d0c3/\n\n\nNow select the URLs for the staged files and print the first one:\n\ns3_urls = [u['href'] for u in response['links'][5:]]\n\nprint(s3_urls[0])\n\ns3://harmony-prod-staging/public/harmony/netcdf-to-zarr/2295236b-8086-4543-9482-f524a9f2d0c3/OCEAN_BOTTOM_PRESSURE_mon_mean_2009-12_ECCO_V4r4_latlon_0p50deg.zarr\n\n\nUse the AWS s3fs package and your temporary aws_creds to open the zarr directory storage:\n\ns3 = s3fs.S3FileSystem(\n key=creds['AccessKeyId'],\n secret=creds['SecretAccessKey'],\n token=creds['SessionToken'],\n client_kwargs={'region_name':'us-west-2'},\n)\n\nlen(s3.ls(s3_dir))\n\n97\n\n\nPlot the first Ocean Bottom Pressure dataset\nCheck out the documentation for xarray’s open_zarr method at this link. Open the first dataset and plot the OBP variable:\n\nds0 = xr.open_zarr(s3.get_mapper(s3_urls[0]), decode_cf=True, mask_and_scale=True)\n\n# Mask the dataset where OBP is not within the bounds of the variable's valid min/max:\nds0_masked = ds0.where((ds0.OBP>=ds0.OBP.valid_min) & (ds0.OBP<=ds0.OBP.valid_max))\n\n# Plot the masked dataset\nds0_masked.OBP.isel(time=0).plot.imshow(size=10)\n\n<matplotlib.image.AxesImage at 0x7f28ed2ba4c0>\n\n\n\n\n\nLoad the zarr datasets into one large xarray dataset\nLoad all the datasets in a loop and concatenate them:\n\nzds = xr.concat([xr.open_zarr(s3.get_mapper(u)) for u in s3_urls], dim=\"time\")\n\nprint(zds)\n\n<xarray.Dataset>\nDimensions: (latitude: 360, longitude: 720, nv: 2, time: 97)\nCoordinates:\n * latitude (latitude) float64 -89.75 -89.25 -88.75 ... 89.25 89.75\n latitude_bnds (latitude, nv) float64 -90.0 -89.5 -89.5 ... 89.5 89.5 90.0\n * longitude (longitude) float64 -179.8 -179.2 -178.8 ... 179.2 179.8\n longitude_bnds (longitude, nv) float64 -180.0 -179.5 -179.5 ... 179.5 180.0\n * time (time) datetime64[ns] 2009-12-16T12:00:00 ... 2017-12-16T...\n time_bnds (time, nv) datetime64[ns] dask.array<chunksize=(1, 2), meta=np.ndarray>\nDimensions without coordinates: nv\nData variables:\n OBP (time, latitude, longitude) float64 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n OBPGMAP (time, latitude, longitude) float64 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\nAttributes: (12/57)\n Conventions: CF-1.8, ACDD-1.3\n acknowledgement: This research was carried out by the Jet Pr...\n author: Ian Fenty and Ou Wang\n cdm_data_type: Grid\n comment: Fields provided on a regular lat-lon grid. ...\n coordinates_comment: Note: the global 'coordinates' attribute de...\n ... ...\n time_coverage_duration: P1M\n time_coverage_end: 2010-01-01T00:00:00\n time_coverage_resolution: P1M\n time_coverage_start: 2009-12-01T00:00:00\n title: ECCO Ocean Bottom Pressure - Monthly Mean 0...\n uuid: 297c8df0-4158-11eb-b208-0cc47a3f687b\n\n\nReference OBP and mask the dataset according to the valid minimum and maximum:\n\nobp = zds.OBP\n\nprint(obp)\n\n<xarray.DataArray 'OBP' (time: 97, latitude: 360, longitude: 720)>\ndask.array<concatenate, shape=(97, 360, 720), dtype=float64, chunksize=(1, 360, 720), chunktype=numpy.ndarray>\nCoordinates:\n * latitude (latitude) float64 -89.75 -89.25 -88.75 ... 88.75 89.25 89.75\n * longitude (longitude) float64 -179.8 -179.2 -178.8 ... 178.8 179.2 179.8\n * time (time) datetime64[ns] 2009-12-16T12:00:00 ... 2017-12-16T06:00:00\nAttributes:\n comment: OBP excludes the contribution from global mean at...\n coverage_content_type: modelResult\n long_name: Ocean bottom pressure given as equivalent water t...\n units: m\n valid_max: 72.07011413574219\n valid_min: -1.7899188995361328\n\n\nGet the valid min and max from the corresponding CF attributes:\n\nobp_vmin, obp_vmax = obp.valid_min, obp.valid_max\n\nobp_vmin, obp_vmax\n\n(-1.7899188995361328, 72.07011413574219)\n\n\nMask the dataset according to the OBP min and max and plot a series:\n\n# Mask dataset where not inside OBP variable valid min/max:\nzds_masked = zds.where((obp>=obp_vmin)&(obp<=obp_vmax))\n\n# Plot SSH again for the first 12 time slices:\nobpp = zds_masked.OBP.isel(time=slice(0, 6)).plot(\n x=\"longitude\", \n y=\"latitude\", \n col=\"time\",\n levels=8,\n col_wrap=3, \n add_colorbar=False,\n figsize=(14, 8)\n)\n\n# Plot a colorbar on a secondary axis\nmappable = obpp.axes[0][0].collections[0]\ncax = plt.axes([0.05, -0.04, 0.95, 0.04])\ncbar1 = plt.colorbar(mappable, cax=cax, orientation='horizontal')" + "objectID": "how-tos/subset.html#how-do-i-subset-an-opendap-granule-in-the-cloud", + "href": "how-tos/subset.html#how-do-i-subset-an-opendap-granule-in-the-cloud", + "title": "How do I subset data granules?", + "section": "How do I subset an OPeNDAP granule in the cloud?", + "text": "How do I subset an OPeNDAP granule in the cloud?" }, { - "objectID": "external/data_discovery_cmr-stac_api.html", - "href": "external/data_discovery_cmr-stac_api.html", - "title": "Data Discovery with CMR-STAC API", + "objectID": "how-tos/subset.html#how-do-i-subset-a-data-granule-using-xarray", + "href": "how-tos/subset.html#how-do-i-subset-a-data-granule-using-xarray", + "title": "How do I subset data granules?", + "section": "How do I subset a data granule using xarray?", + "text": "How do I subset a data granule using xarray?" + }, + { + "objectID": "external/on-prem_cloud.html", + "href": "external/on-prem_cloud.html", + "title": "Pairing Cloud and non-Cloud Data", "section": "", "text": "imported on: 2023-05-30\nThis notebook was originally developed for the 2021 Cloud Hackathon, and has been updated with most current approaches." }, { - "objectID": "external/data_discovery_cmr-stac_api.html#timing", - "href": "external/data_discovery_cmr-stac_api.html#timing", - "title": "Data Discovery with CMR-STAC API", - "section": "Timing", - "text": "Timing\n\nExercise: 30 min" + "objectID": "external/on-prem_cloud.html#accessing-and-harmonizing-data-located-within-and-outside-of-the-nasa-earthdata-cloud", + "href": "external/on-prem_cloud.html#accessing-and-harmonizing-data-located-within-and-outside-of-the-nasa-earthdata-cloud", + "title": "Pairing Cloud and non-Cloud Data", + "section": "Accessing and harmonizing data located within and outside of the NASA Earthdata Cloud", + "text": "Accessing and harmonizing data located within and outside of the NASA Earthdata Cloud" }, { - "objectID": "external/data_discovery_cmr-stac_api.html#summary", - "href": "external/data_discovery_cmr-stac_api.html#summary", - "title": "Data Discovery with CMR-STAC API", - "section": "Summary", - "text": "Summary\nIn this example we will access the NASA’s Harmonized Landsat Sentinel-2 (HLS) version 2 assets, which are archived in cloud optimized geoTIFF (COG) format in the LP DAAC Cumulus cloud space. The COGs can be used like any other geoTIFF file, but have some added features that make them more efficient within the cloud data access paradigm. These features include: overviews and internal tiling. Below we will demonstrate how to leverage these features.\n\nBut first, what is STAC?\nSpatioTemporal Asset Catalog (STAC) is a specification that provides a common language for interpreting geospatial information in order to standardize indexing and discovering data.\nThe STAC specification is made up of a collection of related, yet independent specifications that when used together provide search and discovery capabilities for remote assets.\n\nFour STAC Specifications\nSTAC Catalog (aka DAAC Archive)\nSTAC Collection (aka Data Product)\nSTAC Item (aka Granule)\nSTAC API\nIn the following sections, we will explore each of STAC element using NASA’s Common Metadata Repository (CMR) STAC application programming interface (API), or CMR-STAC API for short.\n\n\n\nCMR-STAC API\nThe CMR-STAC API is NASA’s implementation of the STAC API specification for all NASA data holdings within EOSDIS. The current implementation does not allow for querries accross the entire NASA catalog. Users must execute searches within provider catalogs (e.g., LPCLOUD) to find the STAC Items they are searching for. All the providers can be found at the CMR-STAC endpoint here: https://cmr.earthdata.nasa.gov/stac/.\nIn this exercise, we will query the LPCLOUD provider to identify STAC Items from the Harmonized Landsat Sentinel-2 (HLS) collection that fall within our region of interest (ROI) and within our specified time range." + "objectID": "external/on-prem_cloud.html#timing", + "href": "external/on-prem_cloud.html#timing", + "title": "Pairing Cloud and non-Cloud Data", + "section": "Timing", + "text": "Timing\n\nExercise: 45 min" }, { - "objectID": "external/data_discovery_cmr-stac_api.html#what-you-will-learn-from-this-tutorial", - "href": "external/data_discovery_cmr-stac_api.html#what-you-will-learn-from-this-tutorial", - "title": "Data Discovery with CMR-STAC API", - "section": "What you will learn from this tutorial", - "text": "What you will learn from this tutorial\n\nhow to connect to NASA CMR-STAC API using Python’s pystac-client\n\nhow to navigate CMR-STAC records\n\nhow to read in a geojson file using geopandas to specify your region of interest\nhow to use the CMR-STAC API to search for data\nhow to perform post-search filtering of CMR-STAC API search result in Python\n\nhow to extract and save data access URLs for geospatial assets\n\nThis exercise can be found in the 2021 Cloud Hackathon Book" + "objectID": "external/on-prem_cloud.html#summary", + "href": "external/on-prem_cloud.html#summary", + "title": "Pairing Cloud and non-Cloud Data", + "section": "Summary", + "text": "Summary\nThis tutorial will combine several workflow steps and components from the previous days, demonstrating the process of using the geolocation of data available outside of the Earthdata Cloud to then access coincident variables of cloud-accessible data. This may be a common use case as NASA Earthdata continues to migrate to the cloud, producing a “hybrid” data archive across Amazon Web Services (AWS) and original on-premise data storage systems. Additionally, you may also want to combine field measurements with remote sensing data available on the Earthdata Cloud.\nThis specific example explores the pairing of the ICESat-2 ATL07 Sea Ice Height data product, currently (as of November 2021) available publicly via direct download at the NSIDC DAAC, along with Sea Surface Temperature (SST) from the GHRSST MODIS L2 dataset (MODIS_A-JPL-L2P-v2019.0) available from PO.DAAC on the Earthdata Cloud.\nThe use case we’re looking at today centers over an area north of Greenland for a single day in June, where a melt pond was observed using the NASA OpenAltimetry application. Melt ponds are an important feature of Arctic sea ice dynamics, leading to an decrease in sea ice albedo and other changes in heat balance. Many NASA Earthdata datasets produce variables including sea ice albedo, sea surface temperature, air temperature, and sea ice height, which can be used to better understand these dynamics.\n\nObjectives\n\nPractice skills searching for data in CMR, determining granule coverage across two datasets over an area of interest.\nDownload data from an on-premise storage system to our cloud environment.\nRead in 1-dimensional trajectory data (ICESat-2 ATL07) into xarray and perform attribute conversions.\nSelect and read in sea surface temperature (SST) data (MODIS_A-JPL-L2P-v2019.0) from the Earthdata Cloud into xarray.\nExtract, resample, and plot coincident SST data based on ICESat-2 geolocation." }, { - "objectID": "external/data_discovery_cmr-stac_api.html#import-required-packages", - "href": "external/data_discovery_cmr-stac_api.html#import-required-packages", - "title": "Data Discovery with CMR-STAC API", - "section": "Import Required Packages", - "text": "Import Required Packages\n\nfrom pystac_client import Client \nfrom collections import defaultdict \nimport json\nimport geopandas\nimport geoviews as gv\nfrom cartopy import crs\ngv.extension('bokeh', 'matplotlib')" + "objectID": "external/on-prem_cloud.html#import-packages", + "href": "external/on-prem_cloud.html#import-packages", + "title": "Pairing Cloud and non-Cloud Data", + "section": "Import packages", + "text": "Import packages\n\nimport os\nfrom pathlib import Path\nfrom pprint import pprint\n\n# Access EDS\nimport requests\n\n# Access AWS S3\nimport s3fs\n\n# Read and work with datasets\nimport xarray as xr\nimport numpy as np\nimport h5py\n\n# For plotting\nimport matplotlib.pyplot as plt\nimport cartopy.crs as ccrs\nfrom shapely.geometry import box\n\n# For resampling\nimport pyresample" }, { - "objectID": "external/data_discovery_cmr-stac_api.html#explored-available-nasa-providers", - "href": "external/data_discovery_cmr-stac_api.html#explored-available-nasa-providers", - "title": "Data Discovery with CMR-STAC API", - "section": "Explored available NASA Providers", - "text": "Explored available NASA Providers\n\nSTAC_URL = 'https://cmr.earthdata.nasa.gov/stac'\n\n\nConnect to the CMR-STAC API\n\nprovider_cat = Client.open(STAC_URL)\n\nWe’ll create a providers variable so we can take a deeper look into available data providers - subcategories are referred to as “children”. We can then print them as a for loop.\n\nproviders = [p for p in provider_cat.get_children()]\n\nfor count, provider in enumerate(providers):\n print(f'{count} - {provider.title}')\n\n0 - LARC_ASDC\n1 - USGS_EROS\n2 - ESA\n3 - GHRC\n4 - LAADS\n5 - OBPG\n6 - OB_DAAC\n7 - ECHO\n8 - ISRO\n9 - LPCUMULUS\n10 - EDF_DEV04\n11 - GES_DISC\n12 - ASF\n13 - OMINRT\n14 - EUMETSAT\n15 - NCCS\n16 - NSIDCV0\n17 - PODAAC\n18 - LARC\n19 - USGS\n20 - SCIOPS\n21 - LANCEMODIS\n22 - CDDIS\n23 - JAXA\n24 - AU_AADC\n25 - ECHO10_OPS\n26 - LPDAAC_ECS\n27 - NSIDC_ECS\n28 - ORNL_DAAC\n29 - LM_FIRMS\n30 - SEDAC\n31 - LANCEAMSR2\n32 - NOAA_NCEI\n33 - USGS_LTA\n34 - GESDISCCLD\n35 - GHRSSTCWIC\n36 - ASIPS\n37 - ESDIS\n38 - POCLOUD\n39 - NSIDC_CPRD\n40 - ORNL_CLOUD\n41 - FEDEO\n42 - XYZ_PROV\n43 - GHRC_DAAC\n44 - CSDA\n45 - NRSCC\n46 - CEOS_EXTRA\n47 - MOPITT\n48 - GHRC_CLOUD\n49 - LPCLOUD\n50 - CCMEO" + "objectID": "external/on-prem_cloud.html#specify-data-time-range-and-area-of-interest", + "href": "external/on-prem_cloud.html#specify-data-time-range-and-area-of-interest", + "title": "Pairing Cloud and non-Cloud Data", + "section": "Specify data, time range, and area of interest", + "text": "Specify data, time range, and area of interest\nWe are going to focus on getting data for an area north of Greenland for a single day in June.\nThese bounding_box and temporal variables will be used for data search, subset, and access below:\n\n# Bounding Box spatial parameter in decimal degree 'W,S,E,N' format.\nbounding_box = '-62.8,81.7,-56.4,83'\n\n# Each date in yyyy-MM-ddTHH:mm:ssZ format; date range in start,end format\ntemporal = '2019-06-22T00:00:00Z,2019-06-22T23:59:59Z'\n\nSince we’ve already demonstrated how to locate a dataset’s collection_id and use the cloud_hosted parameter to determine whether a dataset resides in the Earthdata Cloud, we are going to skip forward and declare these variables:\n\nmodis_concept_id = 'C1940473819-POCLOUD'\nicesat2_concept_id = 'C2003771980-NSIDC_ECS'" }, { - "objectID": "external/data_discovery_cmr-stac_api.html#connect-to-the-lpcloud-providerstac-catalog", - "href": "external/data_discovery_cmr-stac_api.html#connect-to-the-lpcloud-providerstac-catalog", - "title": "Data Discovery with CMR-STAC API", - "section": "Connect to the LPCLOUD Provider/STAC Catalog", - "text": "Connect to the LPCLOUD Provider/STAC Catalog\nFor this next step we need the provider title (e.g., LPCLOUD) from above. We will add the provider to the end of the CMR-STAC API URL (i.e., https://cmr.earthdata.nasa.gov/stac/) to connect to the LPCLOUD STAC Catalog.\n\ncatalog = Client.open(f'{STAC_URL}/LPCLOUD/')\n\nSince we are using a dedicated client (i.e., pystac-client.Client) to connect to our STAC Provider Catalog, we will have access to some useful internal methods and functions (e.g., get_children() or get_all_items()) we can use to get information from these objects." + "objectID": "external/on-prem_cloud.html#search-and-download-icesat-2-atl07-files", + "href": "external/on-prem_cloud.html#search-and-download-icesat-2-atl07-files", + "title": "Pairing Cloud and non-Cloud Data", + "section": "Search and download ICESat-2 ATL07 files", + "text": "Search and download ICESat-2 ATL07 files\nPerform a granule search over our time and area of interest. How many granules are returned?\n\ngranule_url = 'https://cmr.earthdata.nasa.gov/search/granules'\n\n\nresponse = requests.get(granule_url,\n params={\n 'concept_id': icesat2_concept_id,\n 'temporal': temporal,\n 'bounding_box': bounding_box,\n 'page_size': 200,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\nprint(response.headers['CMR-Hits'])\n\n2\n\n\nPrint the file names, size, and links:\n\ngranules = response.json()['feed']['entry']\nfor granule in granules:\n print(f'{granule[\"producer_granule_id\"]} {granule[\"granule_size\"]} {granule[\"links\"][0][\"href\"]}')\n\nATL07-01_20190622055317_12980301_004_01.h5 237.0905504227 https://n5eil01u.ecs.nsidc.org/DP9/ATLAS/ATL07.004/2019.06.22/ATL07-01_20190622055317_12980301_004_01.h5\nATL07-01_20190622200154_13070301_004_01.h5 230.9151573181 https://n5eil01u.ecs.nsidc.org/DP9/ATLAS/ATL07.004/2019.06.22/ATL07-01_20190622200154_13070301_004_01.h5\n\n\n\nDownload ATL07 files\nAlthough several services are supported for ICESat-2 data, we are demonstrating direct access through the “on-prem” file system at NSIDC for simplicity.\nSome of these services include: - icepyx - From the icepyx documentation: “icepyx is both a software library and a community composed of ICESat-2 data users, developers, and the scientific community. We are working together to develop a shared library of resources - including existing resources, new code, tutorials, and use-cases/examples - that simplify the process of querying, obtaining, analyzing, and manipulating ICESat-2 datasets to enable scientific discovery.” - NSIDC DAAC Data Access and Service API - The API provided by the NSIDC DAAC allows you to access data programmatically using specific temporal and spatial filters. The same subsetting, reformatting, and reprojection services available on select data sets through NASA Earthdata Search can also be applied using this API. - IceFlow - The IceFlow python library simplifies accessing and combining data from several of NASA’s cryospheric altimetry missions, including ICESat/GLAS, Operation IceBridge, and ICESat-2. In particular, IceFlow harmonizes the various file formats and georeferencing parameters across several of the missions’ data sets, allowing you to analyze data across the multi-decadal time series.\nWe’ve found 2 granules. We’ll download the first one and write it to a file with the same name as the producer_granule_id.\nWe need the url for the granule as well. This is href links we printed out above.\n\nicesat_id = granules[0]['producer_granule_id']\nicesat_url = granules[0]['links'][0]['href']\n\nTo retrieve the granule data, we use the requests.get() method, which will utilize the .netrc file on the backend to authenticate the request against Earthdata Login.\n\nr = requests.get(icesat_url)\n\nThe response returned by requests has the same structure as all the other responses: a header and contents. The header information has information about the response, including the size of the data we downloaded in bytes.\n\nfor k, v in r.headers.items():\n print(f'{k}: {v}')\n\nDate: Thu, 18 Nov 2021 04:02:03 GMT\nServer: Apache\nVary: User-Agent\nContent-Disposition: attachment\nContent-Length: 248607461\nKeep-Alive: timeout=15, max=100\nConnection: Keep-Alive\n\n\nThe contents needs to be saved to a file. To keep the directory clean, we will create a downloads directory to store the file. We can use a shell command to do this or use the makedirs method from the os package.\n\nos.makedirs(\"downloads\", exist_ok=True)\n\nYou should see a downloads directory in the file browser.\nTo write the data to a file, we use open to open a file. We need to specify that the file is open for writing by using the write-mode w. We also need to specify that we want to write bytes by setting the binary-mode b. This is important because the response contents are bytes. The default mode for open is text-mode. So make sure you use b.\nWe’ll use the with statement context-manager to open the file, write the contents of the response, and then close the file. Once the data in r.content is written sucessfully to the file, or if there is an error, the file is closed by the context-manager.\nWe also need to prepend the downloads path to the filename. We do this using Path from the pathlib package in the standard library.\n\noutfile = Path('downloads', icesat_id)\n\n\nif not outfile.exists():\n with open(outfile, 'wb') as f:\n f.write(r.content)\n\nATL07-01_20190622055317_12980301_004_01.h5 is an HDF5 file. xarray can open this but you need to tell it which group to read the data from. In this case we read the sea ice segment height data for ground-track 1 left-beam. You can explore the variable hierarchy in Earthdata Search, by selecting the Customize option under Download Data.\nThis code block performs the following operations: - Extracts the height_segment_height variable from the heights group, along with the dimension variables contained in the higher level sea_ice_segments group, - Convert attributes from bytestrings to strings, - Drops the HDF attribute DIMENSION_LIST, - Sets _FillValue to NaN\n\nvariable_names = [\n '/gt1l/sea_ice_segments/latitude',\n '/gt1l/sea_ice_segments/longitude',\n '/gt1l/sea_ice_segments/delta_time',\n '/gt1l/sea_ice_segments/heights/height_segment_height'\n ]\nwith h5py.File(outfile, 'r') as h5:\n data_vars = {}\n for varname in variable_names:\n var = h5[varname]\n name = varname.split('/')[-1]\n # Convert attributes\n attrs = {}\n for k, v in var.attrs.items():\n if k != 'DIMENSION_LIST':\n if isinstance(v, bytes):\n attrs[k] = v.decode('utf-8')\n else:\n attrs[k] = v\n data = var[:]\n if '_FillValue' in attrs:\n data = np.where(data < attrs['_FillValue'], data, np.nan)\n data_vars[name] = (['segment'], data, attrs)\n is2_ds = xr.Dataset(data_vars)\n \nis2_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (segment: 235584)\nDimensions without coordinates: segment\nData variables:\n latitude (segment) float64 82.38 82.38 82.38 ... 72.61 72.61\n longitude (segment) float64 -55.11 -55.11 ... 145.1 145.1\n delta_time (segment) float64 4.642e+07 4.642e+07 ... 4.642e+07\n height_segment_height (segment) float32 nan nan nan ... -0.4335 -0.4463xarray.DatasetDimensions:segment: 235584Coordinates: (0)Data variables: (4)latitude(segment)float6482.38 82.38 82.38 ... 72.61 72.61contentType :referenceInformationcoordinates :delta_time longitudedescription :Latitude, WGS84, North=+, Lat of segment centerlong_name :Latitudesource :ATBD, section 4.4standard_name :latitudeunits :degrees_northvalid_max :90.0valid_min :-90.0array([82.38431982, 82.38431982, 82.38431982, ..., 72.60984638,\n 72.60977493, 72.60970985])longitude(segment)float64-55.11 -55.11 ... 145.1 145.1contentType :referenceInformationcoordinates :delta_time latitudedescription :Longitude, WGS84, East=+,Lon of segment centerlong_name :Longitudesource :ATBD, section 4.4standard_name :longitudeunits :degrees_eastvalid_max :180.0valid_min :-180.0array([-55.10896068, -55.10896068, -55.10896068, ..., 145.05396164,\n 145.05392851, 145.05389832])delta_time(segment)float644.642e+07 4.642e+07 ... 4.642e+07CLASS :DIMENSION_SCALENAME :gt1l/sea_ice_segments/delta_timeREFERENCE_LIST :[(<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)]contentType :physicalMeasurementcoordinates :latitude longitudedescription :Number of GPS seconds since the ATLAS SDP epoch. The ATLAS Standard Data Products (SDP) epoch offset is defined within /ancillary_data/atlas_sdp_gps_epoch as the number of GPS seconds between the GPS epoch (1980-01-06T00:00:00.000000Z UTC) and the ATLAS SDP epoch. By adding the offset contained within atlas_sdp_gps_epoch to delta time parameters, the time in gps_seconds relative to the GPS epoch can be computed.long_name :Elapsed GPS secondssource :telemetrystandard_name :timeunits :seconds since 2018-01-01array([46419293.64266939, 46419293.64266939, 46419293.64266939, ...,\n 46419681.87646231, 46419681.87759533, 46419681.87862704])height_segment_height(segment)float32nan nan nan ... -0.4335 -0.4463_FillValue :3.4028235e+38contentType :referenceInformationcoordinates :../delta_time ../latitude ../longitudedescription :Mean height from along-track segment fit detremined by the sea ice algorithm. The sea ice height is relative to the tide-free MSS.long_name :height of segment surfacesource :ATBD, section 4.2.2.4units :metersarray([ nan, nan, nan, ..., -0.46550068,\n -0.43347716, -0.4462675 ], dtype=float32)Attributes: (0)\n\n\n\nis2_ds.height_segment_height.plot() ;" }, { - "objectID": "external/data_discovery_cmr-stac_api.html#list-stac-collections", - "href": "external/data_discovery_cmr-stac_api.html#list-stac-collections", - "title": "Data Discovery with CMR-STAC API", - "section": "List STAC Collections", - "text": "List STAC Collections\nWe’ll create a products variable to view deeper in the STAC Catalog.\n\nproducts = [c for c in catalog.get_children()]\n\n\nPrint one of the STAC Collection records\nTo view the products variable we just created, let’s look at one entry as a dictionary.\n\nproducts[1].to_dict()\n\n{'type': 'Collection',\n 'id': 'HLSL30.v2.0',\n 'stac_version': '1.0.0',\n 'description': 'The Harmonized Landsat and Sentinel-2 (HLS) project provides consistent surface reflectance (SR) and top of atmosphere (TOA) brightness data from the Operational Land Imager (OLI) aboard the joint NASA/USGS Landsat 8 satellite and the Multi-Spectral Instrument (MSI) aboard Europe’s Copernicus Sentinel-2A and Sentinel-2B satellites. The combined measurement enables global observations of the land every 2–3 days at 30-meter (m) spatial resolution. The HLS project uses a set of algorithms to obtain seamless products from OLI and MSI that include atmospheric correction, cloud and cloud-shadow masking, spatial co-registration and common gridding, illumination and view angle normalization, and spectral bandpass adjustment.\\r\\n\\r\\nThe HLSL30 product provides 30-m Nadir Bidirectional Reflectance Distribution Function (BRDF)-Adjusted Reflectance (NBAR) and is derived from Landsat 8 OLI data products. The HLSS30 and HLSL30 products are gridded to the same resolution and Military Grid Reference System ([MGRS](https://hls.gsfc.nasa.gov/products-description/tiling-system/)) tiling system, and thus are “stackable” for time series analysis.\\r\\n\\r\\nThe HLSL30 product is provided in Cloud Optimized GeoTIFF (COG) format, and each band is distributed as a separate file. There are 11 bands included in the HLSL30 product along with one quality assessment (QA) band and four angle bands. See the User Guide for a more detailed description of the individual bands provided in the HLSL30 product.',\n 'links': [{'rel': <RelType.ROOT: 'root'>,\n 'href': 'https://cmr.earthdata.nasa.gov/stac/',\n 'type': <MediaType.JSON: 'application/json'>},\n {'rel': 'items',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0/items',\n 'type': 'application/json',\n 'title': 'Granules in this collection'},\n {'rel': 'about',\n 'href': 'https://cmr.earthdata.nasa.gov/search/concepts/C2021957657-LPCLOUD.html',\n 'type': 'text/html',\n 'title': 'HTML metadata for collection'},\n {'rel': 'via',\n 'href': 'https://cmr.earthdata.nasa.gov/search/concepts/C2021957657-LPCLOUD.json',\n 'type': 'application/json',\n 'title': 'CMR JSON metadata for collection'},\n {'rel': 'child',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0/2020',\n 'type': 'application/json',\n 'title': '2020 catalog'},\n {'rel': 'child',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0/2021',\n 'type': 'application/json',\n 'title': '2021 catalog'},\n {'rel': <RelType.SELF: 'self'>,\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0',\n 'type': <MediaType.JSON: 'application/json'>},\n {'rel': <RelType.PARENT: 'parent'>,\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/',\n 'type': <MediaType.JSON: 'application/json'>}],\n 'stac_extensions': [],\n 'title': 'HLS Landsat Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0',\n 'extent': {'spatial': {'bbox': [[-180, -90, 180, 90]]},\n 'temporal': {'interval': [['2013-05-01T00:00:00Z', None]]}},\n 'license': 'not-provided'}\n\n\n\n\nPrint the STAC Collection ids with their title\nIn the above output, id and title are two elements of interest that we can print for all products using a for loop.\n\nfor p in products: \n print(f\"{p.id}: {p.title}\")\n\nASTGTM.v003: ASTER Global Digital Elevation Model V003\nHLSL30.v2.0: HLS Landsat Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0\nHLSL30.v1.5: HLS Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30 m V1.5\nHLSS30.v1.5: HLS Sentinel-2 Multi-spectral Instrument Surface Reflectance Daily Global 30 m V1.5\nHLSS30.v2.0: HLS Sentinel-2 Multi-spectral Instrument Surface Reflectance Daily Global 30m v2.0" + "objectID": "external/on-prem_cloud.html#determine-the-ghrsst-modis-l2-granules-returned-from-our-time-and-area-of-interest", + "href": "external/on-prem_cloud.html#determine-the-ghrsst-modis-l2-granules-returned-from-our-time-and-area-of-interest", + "title": "Pairing Cloud and non-Cloud Data", + "section": "Determine the GHRSST MODIS L2 granules returned from our time and area of interest", + "text": "Determine the GHRSST MODIS L2 granules returned from our time and area of interest\n\nresponse = requests.get(granule_url, \n params={\n 'concept_id': modis_concept_id,\n 'temporal': temporal,\n 'bounding_box': bounding_box,\n 'page_size': 200,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\nprint(response.headers['CMR-Hits'])\n\n14\n\n\n\ngranules = response.json()['feed']['entry']\nfor granule in granules:\n print(f'{granule[\"title\"]} {granule[\"granule_size\"]} {granule[\"links\"][0][\"href\"]}')\n\n20190622000501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 20.71552562713623 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622000501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622014501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 9.34600830078125E-5 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622014501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622032501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 21.307741165161133 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622032501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622050001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 9.34600830078125E-5 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622050001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622050501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 20.065649032592773 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622050501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622064001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 9.34600830078125E-5 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622064001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622064501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0 18.602201461791992 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622064501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\n20190622064501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 18.665077209472656 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622064501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622082001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 19.782299995422363 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622082001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622100001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 19.13440227508545 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622100001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622113501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 20.3239164352417 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622113501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622114001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 9.34600830078125E-5 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622114001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622163001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 19.257243156433105 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622163001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622181001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 19.93498420715332 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622181001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc" }, { - "objectID": "external/data_discovery_cmr-stac_api.html#search-for-granulesstac-items---set-up-query-parameters-to-submit-to-the-cmr-stac-api", - "href": "external/data_discovery_cmr-stac_api.html#search-for-granulesstac-items---set-up-query-parameters-to-submit-to-the-cmr-stac-api", - "title": "Data Discovery with CMR-STAC API", - "section": "Search for Granules/STAC Items - Set up query parameters to submit to the CMR-STAC API", - "text": "Search for Granules/STAC Items - Set up query parameters to submit to the CMR-STAC API\nWe will define our ROI using a geojson file containing a small polygon feature in western Nebraska, USA. The geojson file is found in the ~/data directory. We’ll also specify the data collections and a time range for our example.\n\nRead in a geojson file\nReading in a geojson file with geopandas will return the geometry of our polygon (our ROI).\nNOTE: If you are running the notebook from the tutorials-templates directory, please use the following path to connect to the geojson file: “../tutorials/data/ne_w_agfields.geojson”\n\nfield = geopandas.read_file('./data/ne_w_agfields.geojson')\nfield\n\n\n\n\n\n\n\n\ngeometry\n\n\n\n\n0\nPOLYGON ((-101.67272 41.04754, -101.65345 41.0...\n\n\n\n\n\n\n\n\n\nVisualize contents of geojson file\nWe can use that geometry to visualize the polygon: here, a square. But wait for it –\n\nfieldShape = field['geometry'][0]\nfieldShape\n\n\n\n\nWe can plot the polygon using the geoviews package that we imported as gv with ‘bokeh’ and ‘matplotlib’ extensions. The following has reasonable width, height, color, and line widths to view our polygon when it is overlayed on a base tile map.\n\nbase = gv.tile_sources.EsriImagery.opts(width=650, height=500)\nfarmField = gv.Polygons(fieldShape).opts(line_color='yellow', line_width=10, color=None)\nbase * farmField\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\nWe will now start to specify the search criteria we are interested in, i.e, the date range, the ROI, and the data collections, that we will pass to the STAC API.\n\n\nExtract the coordinates for the ROI\n\nroi = json.loads(field.to_json())['features'][0]['geometry']\nroi\n\n{'type': 'Polygon',\n 'coordinates': [[[-101.67271614074707, 41.04754380304359],\n [-101.65344715118408, 41.04754380304359],\n [-101.65344715118408, 41.06213891056728],\n [-101.67271614074707, 41.06213891056728],\n [-101.67271614074707, 41.04754380304359]]]}\n\n\nSo, what just happen there? Let’s take a quick detour to break it down.\n\n\n\nSpecify date range\nNext up is to specify our date range using ISO_8601 date formatting.\n\n#date_range = \"2021-05-01T00:00:00Z/2021-08-30T23:59:59Z\" # closed interval\n#date_range = \"2021-05-01T00:00:00Z/..\" # open interval - does not currently work with the CMR-STAC API\ndate_range = \"2021-05/2021-08\"\n\n\n\nSpecify the STAC Collections\nSTAC Collection is synonomous with what we usually consider a NASA data product. Desired STAC Collections are submitted to the search API as a list containing the collection id. We can use the ids that we printed from our products for loop above. Let’s focus on S30 and L30 collections.\n\ncollections = ['HLSL30.v2.0', 'HLSS30.v2.0']\ncollections\n\n['HLSL30.v2.0', 'HLSS30.v2.0']" + "objectID": "external/on-prem_cloud.html#load-data-into-xarray-via-s3-direct-access", + "href": "external/on-prem_cloud.html#load-data-into-xarray-via-s3-direct-access", + "title": "Pairing Cloud and non-Cloud Data", + "section": "Load data into xarray via S3 direct access", + "text": "Load data into xarray via S3 direct access\nOur CMR granule search returned 14 files for our time and area of interest. However, not all granules will be suitable for analysis.\nI’ve identified the image with granule id G1956158784-POCLOUD as a good candidate, this is the 9th granule. In this image, our area of interest is close to nadir. This means that the instantaneous field of view over the area of interest cover a smaller area than at the edge of the image.\nWe are looking for the link for direct download access via s3. This is a url but with a prefix s3://. This happens to be the first href link in the metadata.\nFor a single granule we can cut and paste the s3 link. If we have several granules, the s3 links can be extracted with some simple code.\n\ngranule = granules[9]\n\nfor link in granule['links']:\n if link['href'].startswith('s3://'):\n s3_link = link['href']\n \ns3_link\n\n's3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622100001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc'\n\n\n\nGet S3 credentials\nAs with the previous S3 download tutorials we need credentials to access data from s3: access keys and tokens.\n\ns3_credentials = requests.get('https://archive.podaac.earthdata.nasa.gov/s3credentials').json()\n\nEssentially, what we are doing in this step is to “mount” the s3 bucket as a file system. This allows us to treat the S3 bucket in a similar way to a local file system.\n\ns3_fs = s3fs.S3FileSystem(\n key=s3_credentials[\"accessKeyId\"],\n secret=s3_credentials[\"secretAccessKey\"],\n token=s3_credentials[\"sessionToken\"],\n)\n\n\n\nOpen a s3 file\nNow we have the S3FileSystem set up, we can access the granule. xarray cannot open a S3File directly, so we use the open method for the S3FileSystem to open the granule using the endpoint url we extracted from the metadata. We also have to set the mode='rb'. This opens the granule in read-only mode and in byte-mode. Byte-mode is important. By default, open opens a file as text - in this case it would just be a string of characters - and xarray doesn’t know what to do with that.\nWe then pass the S3File object f to xarray.open_dataset. For this dataset, we also have to set decode_cf=False. This switch tells xarray not to use information contained in variable attributes to generate human readable coordinate variables. Normally, this should work for netcdf files but for this particular cloud-hosted dataset, variable attribute data is not in the form expected by xarray. We’ll fix this.\n\nf = s3_fs.open(s3_link, mode='rb')\nmodis_ds = xr.open_dataset(f, decode_cf=False)\n\nIf you click on the Show/Hide Attributes icon (the first document-like icon to the right of coordinate variable metadata) you can see that attributes are one-element arrays containing bytestrings.\n\nmodis_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (nj: 2030, ni: 1354, time: 1)\nCoordinates:\n * time (time) int32 1214042401\nDimensions without coordinates: nj, ni\nData variables:\n lat (nj, ni) float32 ...\n lon (nj, ni) float32 ...\n sea_surface_temperature (time, nj, ni) int16 ...\n sst_dtime (time, nj, ni) int16 ...\n quality_level (time, nj, ni) int8 ...\n sses_bias (time, nj, ni) int8 ...\n sses_standard_deviation (time, nj, ni) int8 ...\n l2p_flags (time, nj, ni) int16 ...\n chlorophyll_a (time, nj, ni) float32 ...\n K_490 (time, nj, ni) int16 ...\n wind_speed (time, nj, ni) int8 ...\n dt_analysis (time, nj, ni) int8 ...\nAttributes: (12/49)\n Conventions: [b'CF-1.7, ACDD-1.3']\n title: [b'MODIS Aqua L2P SST']\n summary: [b'Sea surface temperature retrievals produce...\n references: [b'GHRSST Data Processing Specification v2r5']\n institution: [b'NASA/JPL/OBPG/RSMAS']\n history: [b'MODIS L2P created at JPL PO.DAAC']\n ... ...\n publisher_email: [b'ghrsst-po@nceo.ac.uk']\n processing_level: [b'L2P']\n cdm_data_type: [b'swath']\n startDirection: [b'Ascending']\n endDirection: [b'Descending']\n day_night_flag: [b'Day']xarray.DatasetDimensions:nj: 2030ni: 1354time: 1Coordinates: (1)time(time)int321214042401long_name :[b'reference time of sst file']standard_name :[b'time']units :[b'seconds since 1981-01-01 00:00:00']comment :[b'time of first sensor observation']coverage_content_type :[b'coordinate']array([1214042401], dtype=int32)Data variables: (12)lat(nj, ni)float32...long_name :[b'latitude']standard_name :[b'latitude']units :[b'degrees_north']_FillValue :[-999.]valid_min :[-90.]valid_max :[90.]comment :[b'geographical coordinates, WGS84 projection']coverage_content_type :[b'coordinate'][2748620 values with dtype=float32]lon(nj, ni)float32...long_name :[b'longitude']standard_name :[b'longitude']units :[b'degrees_east']_FillValue :[-999.]valid_min :[-180.]valid_max :[180.]comment :[b'geographical coordinates, WGS84 projection']coverage_content_type :[b'coordinate'][2748620 values with dtype=float32]sea_surface_temperature(time, nj, ni)int16...long_name :[b'sea surface temperature']standard_name :[b'sea_surface_skin_temperature']units :[b'kelvin']_FillValue :[-32767]valid_min :[-1000]valid_max :[10000]comment :[b'sea surface temperature from thermal IR (11 um) channels']scale_factor :[0.005]add_offset :[273.15]source :[b'NASA and University of Miami']coordinates :[b'lon lat']coverage_content_type :[b'physicalMeasurement'][2748620 values with dtype=int16]sst_dtime(time, nj, ni)int16...long_name :[b'time difference from reference time']units :[b'seconds']_FillValue :[-32768]valid_min :[-32767]valid_max :[32767]comment :[b'time plus sst_dtime gives seconds after 00:00:00 UTC January 1, 1981']coordinates :[b'lon lat']coverage_content_type :[b'referenceInformation'][2748620 values with dtype=int16]quality_level(time, nj, ni)int8...long_name :[b'quality level of SST pixel']_FillValue :[-128]valid_min :[0]valid_max :[5]comment :[b'thermal IR SST proximity confidence value; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value']coordinates :[b'lon lat']flag_values :[0 1 2 3 4 5]flag_meanings :[b'no_data bad_data worst_quality low_quality acceptable_quality best_quality']coverage_content_type :[b'qualityInformation'][2748620 values with dtype=int8]sses_bias(time, nj, ni)int8...long_name :[b'SSES bias error based on proximity confidence flags']units :[b'kelvin']_FillValue :[-128]valid_min :[-127]valid_max :[127]comment :[b'thermal IR SST bias error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value']scale_factor :[0.15748031]add_offset :[0.]coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int8]sses_standard_deviation(time, nj, ni)int8...long_name :[b'SSES standard deviation error based on proximity confidence flags']units :[b'kelvin']_FillValue :[-128]valid_min :[-127]valid_max :[127]comment :[b'thermal IR SST standard deviation error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value']scale_factor :[0.07874016]add_offset :[10.]coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int8]l2p_flags(time, nj, ni)int16...long_name :[b'L2P flags']valid_min :[0]valid_max :[16]comment :[b'These flags can be used to further filter data variables']coordinates :[b'lon lat']flag_meanings :[b'microwave land ice lake river']flag_masks :[ 1 2 4 8 16]coverage_content_type :[b'qualityInformation'][2748620 values with dtype=int16]chlorophyll_a(time, nj, ni)float32...long_name :[b'Chlorophyll Concentration, OC3 Algorithm']units :[b'mg m^-3']_FillValue :[-32767.]valid_min :[0.001]valid_max :[100.]comment :[b'non L2P core field']coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=float32]K_490(time, nj, ni)int16...long_name :[b'Diffuse attenuation coefficient at 490 nm (OBPG)']units :[b'm^-1']_FillValue :[-32767]valid_min :[50]valid_max :[30000]comment :[b'non L2P core field']scale_factor :[0.0002]add_offset :[0.]coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int16]wind_speed(time, nj, ni)int8...long_name :[b'10m wind speed']standard_name :[b'wind_speed']units :[b'm s-1']_FillValue :[-128]valid_min :[-127]valid_max :[127]comment :[b'Wind at 10 meters above the sea surface']scale_factor :[0.2]add_offset :[25.]source :[b'TBD. Placeholder. Currently empty']coordinates :[b'lon lat']grid_mapping :[b'TBD']time_offset :[2.]height :[b'10 m']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int8]dt_analysis(time, nj, ni)int8...long_name :[b'deviation from SST reference climatology']units :[b'kelvin']_FillValue :[-128]valid_min :[-127]valid_max :[127]comment :[b'TBD']scale_factor :[0.1]add_offset :[0.]source :[b'TBD. Placeholder. Currently empty']coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int8]Attributes: (49)Conventions :[b'CF-1.7, ACDD-1.3']title :[b'MODIS Aqua L2P SST']summary :[b'Sea surface temperature retrievals produced at the NASA OBPG for the MODIS Aqua sensor. These have been reformatted to GHRSST GDS specifications by the JPL PO.DAAC']references :[b'GHRSST Data Processing Specification v2r5']institution :[b'NASA/JPL/OBPG/RSMAS']history :[b'MODIS L2P created at JPL PO.DAAC']comment :[b'L2P Core without DT analysis or other ancillary fields; Day, Start Node:Ascending, End Node:Descending; WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value; Refined']license :[b'GHRSST and PO.DAAC protocol allow data use as free and open.']id :[b'MODIS_A-JPL-L2P-v2019.0']naming_authority :[b'org.ghrsst']product_version :[b'2019.0']uuid :[b'f6e1f61d-c4a4-4c17-8354-0c15e12d688b']gds_version_id :[b'2.0']netcdf_version_id :[b'4.1']date_created :[b'20200221T085224Z']file_quality_level :[3]spatial_resolution :[b'1km']start_time :[b'20190622T100001Z']time_coverage_start :[b'20190622T100001Z']stop_time :[b'20190622T100459Z']time_coverage_end :[b'20190622T100459Z']northernmost_latitude :[89.9862]southernmost_latitude :[66.2723]easternmost_longitude :[-45.9467]westernmost_longitude :[152.489]source :[b'MODIS sea surface temperature observations for the OBPG']platform :[b'Aqua']sensor :[b'MODIS']metadata_link :[b'http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MODIS_A-JPL-L2P-v2019.0']keywords :[b'Oceans > Ocean Temperature > Sea Surface Temperature']keywords_vocabulary :[b'NASA Global Change Master Directory (GCMD) Science Keywords']standard_name_vocabulary :[b'NetCDF Climate and Forecast (CF) Metadata Convention']geospatial_lat_units :[b'degrees_north']geospatial_lat_resolution :[0.01]geospatial_lon_units :[b'degrees_east']geospatial_lon_resolution :[0.01]acknowledgment :[b'The MODIS L2P sea surface temperature data are sponsored by NASA']creator_name :[b'Ed Armstrong, JPL PO.DAAC']creator_email :[b'edward.m.armstrong@jpl.nasa.gov']creator_url :[b'http://podaac.jpl.nasa.gov']project :[b'Group for High Resolution Sea Surface Temperature']publisher_name :[b'The GHRSST Project Office']publisher_url :[b'http://www.ghrsst.org']publisher_email :[b'ghrsst-po@nceo.ac.uk']processing_level :[b'L2P']cdm_data_type :[b'swath']startDirection :[b'Ascending']endDirection :[b'Descending']day_night_flag :[b'Day']\n\n\nTo fix this, we need to extract array elements as scalars, and convert those scalars from bytestrings to strings. We use the decode method to do this. The bytestrings are encoded as utf-8, which is a unicode character format. This is the default encoding for decode but we’ve included it as an argument to be explicit.\nNot all attributes are bytestrings. Some are floats. Take a look at _FillValue, and valid_min and valid_max. To avoid an error, we use the isinstance function to check if the value of an attributes is type bytes - a bytestring. If it is, then we decode it. If not, we just extract the scalar and do nothing else.\nWe also fix the global attributes.\n\ndef fix_attributes(da):\n '''Decodes bytestring attributes to strings'''\n for attr, value in da.attrs.items():\n if isinstance(value[0], bytes):\n da.attrs[attr] = value[0].decode('utf-8')\n else:\n da.attrs[attr] = value[0]\n return\n\n# Fix variable attributes\nfor var in modis_ds.variables:\n da = modis_ds[var]\n fix_attributes(da)\n \n# Fix global attributes\nfix_attributes(modis_ds)\n\nWith this done, we can use the xarray function decode_cf to convert the attributes.\n\nmodis_ds = xr.decode_cf(modis_ds)\n\n\nmodis_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (nj: 2030, ni: 1354, time: 1)\nCoordinates:\n lat (nj, ni) float32 ...\n lon (nj, ni) float32 ...\n * time (time) datetime64[ns] 2019-06-22T10:00:01\nDimensions without coordinates: nj, ni\nData variables:\n sea_surface_temperature (time, nj, ni) float32 ...\n sst_dtime (time, nj, ni) timedelta64[ns] ...\n quality_level (time, nj, ni) float32 ...\n sses_bias (time, nj, ni) float32 ...\n sses_standard_deviation (time, nj, ni) float32 ...\n l2p_flags (time, nj, ni) int16 ...\n chlorophyll_a (time, nj, ni) float32 ...\n K_490 (time, nj, ni) float32 ...\n wind_speed (time, nj, ni) float32 ...\n dt_analysis (time, nj, ni) float32 ...\nAttributes: (12/49)\n Conventions: CF-1.7, ACDD-1.3\n title: MODIS Aqua L2P SST\n summary: Sea surface temperature retrievals produced a...\n references: GHRSST Data Processing Specification v2r5\n institution: NASA/JPL/OBPG/RSMAS\n history: MODIS L2P created at JPL PO.DAAC\n ... ...\n publisher_email: ghrsst-po@nceo.ac.uk\n processing_level: L2P\n cdm_data_type: swath\n startDirection: Ascending\n endDirection: Descending\n day_night_flag: Dayxarray.DatasetDimensions:nj: 2030ni: 1354time: 1Coordinates: (3)lat(nj, ni)float32...long_name :latitudestandard_name :latitudeunits :degrees_northvalid_min :-90.0valid_max :90.0comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[2748620 values with dtype=float32]lon(nj, ni)float32...long_name :longitudestandard_name :longitudeunits :degrees_eastvalid_min :-180.0valid_max :180.0comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[2748620 values with dtype=float32]time(time)datetime64[ns]2019-06-22T10:00:01long_name :reference time of sst filestandard_name :timecomment :time of first sensor observationcoverage_content_type :coordinatearray(['2019-06-22T10:00:01.000000000'], dtype='datetime64[ns]')Data variables: (10)sea_surface_temperature(time, nj, ni)float32...long_name :sea surface temperaturestandard_name :sea_surface_skin_temperatureunits :kelvinvalid_min :-1000valid_max :10000comment :sea surface temperature from thermal IR (11 um) channelssource :NASA and University of Miamicoverage_content_type :physicalMeasurement[2748620 values with dtype=float32]sst_dtime(time, nj, ni)timedelta64[ns]...long_name :time difference from reference timevalid_min :-32767valid_max :32767comment :time plus sst_dtime gives seconds after 00:00:00 UTC January 1, 1981coverage_content_type :referenceInformation[2748620 values with dtype=timedelta64[ns]]quality_level(time, nj, ni)float32...long_name :quality level of SST pixelvalid_min :0valid_max :5comment :thermal IR SST proximity confidence value; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valueflag_values :0flag_meanings :no_data bad_data worst_quality low_quality acceptable_quality best_qualitycoverage_content_type :qualityInformation[2748620 values with dtype=float32]sses_bias(time, nj, ni)float32...long_name :SSES bias error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :thermal IR SST bias error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]sses_standard_deviation(time, nj, ni)float32...long_name :SSES standard deviation error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :thermal IR SST standard deviation error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]l2p_flags(time, nj, ni)int16...long_name :L2P flagsvalid_min :0valid_max :16comment :These flags can be used to further filter data variablesflag_meanings :microwave land ice lake riverflag_masks :1coverage_content_type :qualityInformation[2748620 values with dtype=int16]chlorophyll_a(time, nj, ni)float32...long_name :Chlorophyll Concentration, OC3 Algorithmunits :mg m^-3valid_min :0.001valid_max :100.0comment :non L2P core fieldcoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]K_490(time, nj, ni)float32...long_name :Diffuse attenuation coefficient at 490 nm (OBPG)units :m^-1valid_min :50valid_max :30000comment :non L2P core fieldcoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]wind_speed(time, nj, ni)float32...long_name :10m wind speedstandard_name :wind_speedunits :m s-1valid_min :-127valid_max :127comment :Wind at 10 meters above the sea surfacesource :TBD. Placeholder. Currently emptygrid_mapping :TBDtime_offset :2.0height :10 mcoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]dt_analysis(time, nj, ni)float32...long_name :deviation from SST reference climatologyunits :kelvinvalid_min :-127valid_max :127comment :TBDsource :TBD. Placeholder. Currently emptycoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]Attributes: (49)Conventions :CF-1.7, ACDD-1.3title :MODIS Aqua L2P SSTsummary :Sea surface temperature retrievals produced at the NASA OBPG for the MODIS Aqua sensor. These have been reformatted to GHRSST GDS specifications by the JPL PO.DAACreferences :GHRSST Data Processing Specification v2r5institution :NASA/JPL/OBPG/RSMAShistory :MODIS L2P created at JPL PO.DAACcomment :L2P Core without DT analysis or other ancillary fields; Day, Start Node:Ascending, End Node:Descending; WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value; Refinedlicense :GHRSST and PO.DAAC protocol allow data use as free and open.id :MODIS_A-JPL-L2P-v2019.0naming_authority :org.ghrsstproduct_version :2019.0uuid :f6e1f61d-c4a4-4c17-8354-0c15e12d688bgds_version_id :2.0netcdf_version_id :4.1date_created :20200221T085224Zfile_quality_level :3spatial_resolution :1kmstart_time :20190622T100001Ztime_coverage_start :20190622T100001Zstop_time :20190622T100459Ztime_coverage_end :20190622T100459Znorthernmost_latitude :89.9862southernmost_latitude :66.2723easternmost_longitude :-45.9467westernmost_longitude :152.489source :MODIS sea surface temperature observations for the OBPGplatform :Aquasensor :MODISmetadata_link :http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MODIS_A-JPL-L2P-v2019.0keywords :Oceans > Ocean Temperature > Sea Surface Temperaturekeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordsstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventiongeospatial_lat_units :degrees_northgeospatial_lat_resolution :0.01geospatial_lon_units :degrees_eastgeospatial_lon_resolution :0.01acknowledgment :The MODIS L2P sea surface temperature data are sponsored by NASAcreator_name :Ed Armstrong, JPL PO.DAACcreator_email :edward.m.armstrong@jpl.nasa.govcreator_url :http://podaac.jpl.nasa.govproject :Group for High Resolution Sea Surface Temperaturepublisher_name :The GHRSST Project Officepublisher_url :http://www.ghrsst.orgpublisher_email :ghrsst-po@nceo.ac.ukprocessing_level :L2Pcdm_data_type :swathstartDirection :AscendingendDirection :Descendingday_night_flag :Day\n\n\nLet’s make a quick plot to take a look at the sea_surface_temperature variable.\n\nmodis_ds.sea_surface_temperature.plot() ;\n\n\n\n\n\n\nPlot MODIS and ICESat-2 data on a map\n\nmap_proj = ccrs.NorthPolarStereo()\n\nfig = plt.figure(figsize=(10,5))\nax = fig.add_subplot(projection=map_proj)\nax.coastlines()\n\n# Plot MODIS sst, save object as sst_img, so we can add colorbar\nsst_img = ax.pcolormesh(modis_ds.lon, modis_ds.lat, modis_ds.sea_surface_temperature[0,:,:], \n vmin=240, vmax=270, # Set max and min values for plotting\n cmap='viridis', shading='auto', # shading='auto' to avoid warning\n transform=ccrs.PlateCarree()) # coords are lat,lon but map if NPS \n\n# Plot IS2 surface height \nis2_img = ax.scatter(is2_ds.longitude, is2_ds.latitude,\n c=is2_ds.height_segment_height, \n vmax=1.5, # Set max height to plot\n cmap='Reds', alpha=0.6, s=2,\n transform=ccrs.PlateCarree())\n\n# Add colorbars\nfig.colorbar(sst_img, label='MODIS SST (K)')\nfig.colorbar(is2_img, label='ATL07 Height (m)')\n\n\n<matplotlib.colorbar.Colorbar at 0x7fd2eda24580>\n\n\n\n\n\n\n\nExtract SST coincident with ICESat-2 track\nThe MODIS SST is swath data, not a regularly-spaced grid of sea surface temperatures. ICESat-2 sea surface heights are irregularly spaced segments along one ground-track traced by the ATLAS instrument on-board ICESat-2. Fortunately, pyresample allows us to resample swath data.\npyresample has many resampling methods. We’re going to use the nearest neighbour resampling method, which is implemented using a k-dimensional tree algorithm or K-d tree. K-d trees are data structures that improve search efficiency for large data sets.\nThe first step is to define the geometry of the ICESat-2 and MODIS data. To do this we use the latitudes and longitudes of the datasets.\n\nis2_geometry = pyresample.SwathDefinition(lons=is2_ds.longitude,\n lats=is2_ds.latitude)\n\n\nmodis_geometry = pyresample.SwathDefinition(lons=modis_ds.lon, lats=modis_ds.lat)\n\nWe then implement the resampling method, passing the two geometries we have defined, the data array we want to resample - in this case sea surface temperature, and a search radius. The resampling method expects a numpy.Array rather than an xarray.DataArray, so we use values to get the data as a numpy.Array.\nWe set the search radius to 1000 m. The MODIS data is nominally 1km spacing.\n\nsearch_radius=1000.\nfill_value = np.nan\nis2_sst = pyresample.kd_tree.resample_nearest(\n modis_geometry,\n modis_ds.sea_surface_temperature.values,\n is2_geometry,\n search_radius,\n fill_value=fill_value\n)\n\n\nis2_sst\n\narray([263.375, 263.375, 263.375, ..., nan, nan, nan],\n dtype=float32)\n\n\n\nis2_ds['sea_surface_temperature'] = xr.DataArray(is2_sst, dims='segment')\nis2_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (segment: 235584)\nDimensions without coordinates: segment\nData variables:\n latitude (segment) float64 82.38 82.38 82.38 ... 72.61 72.61\n longitude (segment) float64 -55.11 -55.11 ... 145.1 145.1\n delta_time (segment) float64 4.642e+07 4.642e+07 ... 4.642e+07\n height_segment_height (segment) float32 nan nan nan ... -0.4335 -0.4463\n sea_surface_temperature (segment) float32 263.4 263.4 263.4 ... nan nan nanxarray.DatasetDimensions:segment: 235584Coordinates: (0)Data variables: (5)latitude(segment)float6482.38 82.38 82.38 ... 72.61 72.61contentType :referenceInformationcoordinates :delta_time longitudedescription :Latitude, WGS84, North=+, Lat of segment centerlong_name :Latitudesource :ATBD, section 4.4standard_name :latitudeunits :degrees_northvalid_max :90.0valid_min :-90.0array([82.38431982, 82.38431982, 82.38431982, ..., 72.60984638,\n 72.60977493, 72.60970985])longitude(segment)float64-55.11 -55.11 ... 145.1 145.1contentType :referenceInformationcoordinates :delta_time latitudedescription :Longitude, WGS84, East=+,Lon of segment centerlong_name :Longitudesource :ATBD, section 4.4standard_name :longitudeunits :degrees_eastvalid_max :180.0valid_min :-180.0array([-55.10896068, -55.10896068, -55.10896068, ..., 145.05396164,\n 145.05392851, 145.05389832])delta_time(segment)float644.642e+07 4.642e+07 ... 4.642e+07CLASS :DIMENSION_SCALENAME :gt1l/sea_ice_segments/delta_timeREFERENCE_LIST :[(<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)]contentType :physicalMeasurementcoordinates :latitude longitudedescription :Number of GPS seconds since the ATLAS SDP epoch. The ATLAS Standard Data Products (SDP) epoch offset is defined within /ancillary_data/atlas_sdp_gps_epoch as the number of GPS seconds between the GPS epoch (1980-01-06T00:00:00.000000Z UTC) and the ATLAS SDP epoch. By adding the offset contained within atlas_sdp_gps_epoch to delta time parameters, the time in gps_seconds relative to the GPS epoch can be computed.long_name :Elapsed GPS secondssource :telemetrystandard_name :timeunits :seconds since 2018-01-01array([46419293.64266939, 46419293.64266939, 46419293.64266939, ...,\n 46419681.87646231, 46419681.87759533, 46419681.87862704])height_segment_height(segment)float32nan nan nan ... -0.4335 -0.4463_FillValue :3.4028235e+38contentType :referenceInformationcoordinates :../delta_time ../latitude ../longitudedescription :Mean height from along-track segment fit detremined by the sea ice algorithm. The sea ice height is relative to the tide-free MSS.long_name :height of segment surfacesource :ATBD, section 4.2.2.4units :metersarray([ nan, nan, nan, ..., -0.46550068,\n -0.43347716, -0.4462675 ], dtype=float32)sea_surface_temperature(segment)float32263.4 263.4 263.4 ... nan nan nanarray([263.375, 263.375, 263.375, ..., nan, nan, nan],\n dtype=float32)Attributes: (0)\n\n\n\n\nPlot SST and Height along track\nThis is a quick plot of the extracted data. We’re using matplotlib so we can use latitude as the x-value:\n\nis2_ds = is2_ds.set_coords(['latitude'])\n\nfig, ax1 = plt.subplots(figsize=(15, 7))\nax1.set_xlim(82.,88.)\nax1.plot(is2_ds.latitude, is2_ds.sea_surface_temperature, \n color='orange', label='SST', zorder=3)\nax1.set_ylabel('SST (K)')\n\nax2 = ax1.twinx()\nax2.plot(is2_ds.latitude, is2_ds.height_segment_height, label='Height')\nax2.set_ylabel('Height (m)')\n\nfig.legend()\n\n<matplotlib.legend.Legend at 0x7fd2ef2ea880>" }, { - "objectID": "external/data_discovery_cmr-stac_api.html#search-the-cmr-stac-api-with-our-search-criteria", - "href": "external/data_discovery_cmr-stac_api.html#search-the-cmr-stac-api-with-our-search-criteria", - "title": "Data Discovery with CMR-STAC API", - "section": "Search the CMR-STAC API with our search criteria", - "text": "Search the CMR-STAC API with our search criteria\nNow we can put all our search criteria together using catalog.search from the pystac_client package.\n\nsearch = catalog.search(\n collections=collections,\n intersects=roi,\n datetime=date_range,\n limit=100\n)\n\n\nPrint out how many STAC Items match our search query\n\nsearch.matched()\n\n113\n\n\nWe now have a search object containing the STAC Items that matched our query. Now, let’s pull out all of the STAC Items (as a PySTAC ItemCollection object) and explore the contents (i.e., the STAC Items)\n\nitem_collection = search.get_all_items()\n\nLet’s list some of the Items from our pystac item_collection:\n\nlist(item_collection)[0:5]\n\n[<Item id=HLS.L30.T13TGF.2021124T173013.v2.0>,\n <Item id=HLS.L30.T14TKL.2021124T173013.v2.0>,\n <Item id=HLS.S30.T14TKL.2021125T172901.v2.0>,\n <Item id=HLS.S30.T13TGF.2021125T172901.v2.0>,\n <Item id=HLS.S30.T14TKL.2021128T173901.v2.0>]\n\n\nWe can view a single Item as a dictionary, as we did above with STAC Collections/products.\n\nitem_collection[0].to_dict()\n\n{'type': 'Feature',\n 'stac_version': '1.0.0',\n 'id': 'HLS.L30.T13TGF.2021124T173013.v2.0',\n 'properties': {'datetime': '2021-05-04T17:30:13.428000Z',\n 'start_datetime': '2021-05-04T17:30:13.428Z',\n 'end_datetime': '2021-05-04T17:30:37.319Z',\n 'eo:cloud_cover': 36},\n 'geometry': {'type': 'Polygon',\n 'coordinates': [[[-101.5423534, 40.5109845],\n [-101.3056118, 41.2066375],\n [-101.2894253, 41.4919436],\n [-102.6032964, 41.5268623],\n [-102.638891, 40.5386175],\n [-101.5423534, 40.5109845]]]},\n 'links': [{'rel': 'self',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0/items/HLS.L30.T13TGF.2021124T173013.v2.0'},\n {'rel': 'parent',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0'},\n {'rel': 'collection',\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0'},\n {'rel': <RelType.ROOT: 'root'>,\n 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/',\n 'type': <MediaType.JSON: 'application/json'>},\n {'rel': 'provider', 'href': 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD'},\n {'rel': 'via',\n 'href': 'https://cmr.earthdata.nasa.gov/search/concepts/G2144020713-LPCLOUD.json'},\n {'rel': 'via',\n 'href': 'https://cmr.earthdata.nasa.gov/search/concepts/G2144020713-LPCLOUD.umm_json'}],\n 'assets': {'B11': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B11.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B11.tif'},\n 'B07': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B07.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B07.tif'},\n 'SAA': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.SAA.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.SAA.tif'},\n 'B06': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B06.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B06.tif'},\n 'B09': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B09.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B09.tif'},\n 'B10': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B10.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B10.tif'},\n 'VZA': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.VZA.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.VZA.tif'},\n 'SZA': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.SZA.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.SZA.tif'},\n 'B01': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B01.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B01.tif'},\n 'VAA': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.VAA.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.VAA.tif'},\n 'B05': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B05.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B05.tif'},\n 'B02': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B02.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B02.tif'},\n 'Fmask': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.Fmask.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.Fmask.tif'},\n 'B03': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B03.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B03.tif'},\n 'B04': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.B04.tif',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.B04.tif'},\n 'browse': {'href': 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-public/HLSL30.020/HLS.L30.T13TGF.2021124T173013.v2.0/HLS.L30.T13TGF.2021124T173013.v2.0.jpg',\n 'type': 'image/jpeg',\n 'title': 'Download HLS.L30.T13TGF.2021124T173013.v2.0.jpg'},\n 'metadata': {'href': 'https://cmr.earthdata.nasa.gov/search/concepts/G2144020713-LPCLOUD.xml',\n 'type': 'application/xml'}},\n 'bbox': [-102.638891, 40.510984, -101.289425, 41.526862],\n 'stac_extensions': ['https://stac-extensions.github.io/eo/v1.0.0/schema.json'],\n 'collection': 'HLSL30.v2.0'}" + "objectID": "external/cof-zarr-reformat.html", + "href": "external/cof-zarr-reformat.html", + "title": "COF Zarr Access via Reformat", + "section": "", + "text": "imported on: 2023-05-30\nThis notebook is from NASA’s PO.DAAC, Access ECCO data via Harmony and the Zarr reformatter service example" }, { - "objectID": "external/data_discovery_cmr-stac_api.html#filtering-stac-items", - "href": "external/data_discovery_cmr-stac_api.html#filtering-stac-items", - "title": "Data Discovery with CMR-STAC API", - "section": "Filtering STAC Items", - "text": "Filtering STAC Items\nWhile the CMR-STAC API is a powerful search and discovery utility, it is still maturing and currently does not have the full gamut of filtering capabilities that the STAC API specification allows for. Hence, additional filtering is required if we want to filter by a property, for example cloud cover. Below we will loop through and filter the item_collection by a specified cloud cover as well as extract the band we’d need to do an Enhanced Vegetation Index (EVI) calculation for a future analysis.\nWe’ll make a cloudcover variable where we will set the maximum allowable cloud cover and extract the band links for those Items that match or are less than the max cloud cover.\n\ncloudcover = 25\n\nWe will also specify the STAC Assets (i.e., bands/layers) of interest for both the S30 and L30 collections (also in our collections variable above).\nIn this hypothetical workflow, we’ll extract the bands needed to calculate an enhanced vegetation index (EVI). Thus, the band needed include red, near infrared (NIR), and blue. We’ll also extract a quality band (i.e., Fmask) that we’d eventually use to perform per-pixel quality filtering.\nNotice that the band ids are in some case not one-to-one between the S30 and the L30 product. This is evident in the NIR band for each product where S30’s NIR band id is B8A and L30’s is B05. Note, the S30 product has an additional NIR band with a band id of B08, but the spectral ranges between B8A and B05 are more closely aligned. Visit the HLS Overview page to learn more about HLS spectral bands.\n\ns30_bands = ['B8A', 'B04', 'B02', 'Fmask'] # S30 bands for EVI calculation and quality filtering -> NIR, RED, BLUE, Quality \nl30_bands = ['B05', 'B04', 'B02', 'Fmask'] # L30 bands for EVI calculation and quality filtering -> NIR, RED, BLUE, Quality \n\nAnd now to loop through and filter the item_collection by cloud cover and bands:\n\nevi_band_links = []\n\nfor i in item_collection:\n if i.properties['eo:cloud_cover'] <= cloudcover:\n if i.collection_id == 'HLSS30.v2.0':\n #print(i.properties['eo:cloud_cover'])\n evi_bands = s30_bands\n elif i.collection_id == 'HLSL30.v2.0':\n #print(i.properties['eo:cloud_cover'])\n evi_bands = l30_bands\n\n for a in i.assets:\n if any(b==a for b in evi_bands):\n evi_band_links.append(i.assets[a].href)\n\nThe filtering done in the previous steps produces a list of links to STAC Assets. Let’s print out the first ten links.\n\nevi_band_links[:10]\n\n['https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B05.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.Fmask.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B02.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T14TKL.2021133T172406.v2.0/HLS.L30.T14TKL.2021133T172406.v2.0.B02.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T14TKL.2021133T172406.v2.0/HLS.L30.T14TKL.2021133T172406.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T14TKL.2021133T172406.v2.0/HLS.L30.T14TKL.2021133T172406.v2.0.B05.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T14TKL.2021133T172406.v2.0/HLS.L30.T14TKL.2021133T172406.v2.0.Fmask.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T14TKL.2021133T173859.v2.0/HLS.S30.T14TKL.2021133T173859.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T14TKL.2021133T173859.v2.0/HLS.S30.T14TKL.2021133T173859.v2.0.B8A.tif']\n\n\nNOTE that HLS data is mapped to the Universal Transverse Mercator (UTM) projection and is tiled using the Sentinel-2 Military Grid Reference System (MGRS) UTM grid. Notice that in the list of links we have multiple tiles, i.e. T14TKL & T13TGF, that intersect with our region of interest. In this case, these two tiles represent neighboring UTM zones. The tiles can be discern from the file name, which is the last element in a link (far right) following the last forward slash (/) - e.g., HLS.L30.T14TKL.2021133T172406.v1.5.B04.tif. The figure below explains where to find the tile/UTM zone from the file name.\n\nWe will now split the list of links into separate logical sub-lists." + "objectID": "external/cof-zarr-reformat.html#getting-started", + "href": "external/cof-zarr-reformat.html#getting-started", + "title": "COF Zarr Access via Reformat", + "section": "Getting Started", + "text": "Getting Started\nWe will access monthly ocean bottom pressure (OBP) data from ECCO V4r4 (10.5067/ECG5M-OBP44), which are provided as a monthly time series on a 0.5-degree latitude/longitude grid.\nThe data are archived in netCDF format. However, this notebook demonstration will request conversion to Zarr format for files covering the period between 2010 and 2018. Upon receiving our request, Harmony’s backend will convert the files and stage them in S3 for native access in AWS (us-west-2 region, specifically). We will access the new Zarr datasets as an aggregated dataset using xarray, and leverage the S3 native protocols for direct access to the data in an efficient manner.\n\n\nRequirements\n\nAWS\nThis notebook should be running in an EC2 instance in AWS region us-west-2, as previously mentioned. We recommend using an EC2 with at least 8GB of memory available.\nThe notebook was developed and tested using a t2.large instance (2 cpus; 8GB memory).\n\n\nPython 3\nMost of these imports are from the Python standard library. However, you will need to install these packages into your Python 3 environment if you have not already done so:\n\ns3fs\nrequests\npandas\nxarray\nmatplotlib\n\n\n\n\nRequirements\n\nimport matplotlib.pyplot as plt\nimport xarray as xr\nimport pandas as pd\nimport numpy as np\nimport requests\nimport json\nimport time\nimport s3fs\n\nShortName = \"ECCO_L4_OBP_05DEG_MONTHLY_V4R4\"\n\n\n\nStudy period\nSet some “master” inputs to define the time and place contexts for our case studies in the ipynb. This example will be requesting time subsets and receiving global data back from the Harmony API.\n\nstart_date = \"2010-01-01\"\nend_date = \"2018-12-31\"\n\n\n\nData Access\nSome features in the Harmony API require us to identify the target dataset/collection by its concept-id (which uniquely idenfifies it among the other datasets in the Common Metadata Repository). Support for selection by the dataset ShortName will be added in a future release.\n\nCommon Metadata Repository (CMR)\nFor now, we will need to get the concept-id that corresponds to our dataset by accessing its metadata from the CMR. Read more about the CMR at: https://cmr.earthdata.nasa.gov/\nRequest the UMM Collection metadata (i.e. metadata about the dataset) from the CMR and select the concept-id as a new variable ccid.\n\nresponse = requests.get(\n url='https://cmr.earthdata.nasa.gov/search/collections.umm_json', \n params={'provider': \"POCLOUD\",\n 'ShortName': ShortName,\n 'page_size': 1}\n)\n\nummc = response.json()['items'][0]\n\nccid = ummc['meta']['concept-id']\n\nccid\n\n'C1990404791-POCLOUD'\n\n\n\n\nHarmony API\nAnd get the Harmony API endpoint and zarr parameter like we did for SMAP before:\n\nbase = f\"https://harmony.earthdata.nasa.gov/{ccid}\"\nhreq = f\"{base}/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset\"\nrurl = f\"{hreq}?format=application/x-zarr\"\n\nprint(rurl)\n\nhttps://harmony.earthdata.nasa.gov/C1990404791-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?format=application/x-zarr\n\n\nECCO monthly collections have 312 granules in V4r4 (you can confirm with the granule listing from CMR Search API) so we can get the entire time series for 2010 to 2018 with one request to the Harmony API.\nFormat a string of query parameters to limit the processing to the desired time period. Then, append the string of time subset parameters to the variable rurl.\n\nsubs = '&'.join([f'subset=time(\"{start_date}T00:00:00.000Z\":\"{end_date}T23:59:59.999Z\")'])\n\nrurl = f\"{rurl}&{subs}\"\n\nprint(rurl)\n\nhttps://harmony.earthdata.nasa.gov/C1990404791-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?format=application/x-zarr&subset=time(\"2010-01-01T00:00:00.000Z\":\"2018-12-31T23:59:59.999Z\")\n\n\nSubmit the request and monitor the processing status in a while loop, breaking it on completion of the request job:\n\nresponse = requests.get(url=rurl).json()\n\n# Monitor status in a while loop. Wait 10 seconds for each check.\nwait = 10\nwhile True:\n response = requests.get(url=response['links'][0]['href']).json()\n if response['status']!='running':\n break\n print(f\"Job in progress ({response['progress']}%)\")\n time.sleep(wait)\n\nprint(\"DONE!\")\n\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nJob in progress (0%)\nDONE!\n\n\nAccess the staged cloud datasets over native AWS interfaces\nCheck the message field in the response for clues about how to proceed:\n\nprint(response['message'])\n\nThe job has completed successfully. Contains results in AWS S3. Access from AWS us-west-2 with keys from https://harmony.earthdata.nasa.gov/cloud-access.sh\n\n\nThe third item in the list of links contains the shell script from the job status message printed above. Let’s download the same information in JSON format. It should be the fourth item; check to be sure:\n\nlen(response['links'])\n\n102\n\n\nSelect the url and download the json, then load to Python dictionary and print the keys:\n\nwith requests.get(response['links'][3]['href']) as r:\n creds = r.json()\n\nprint(creds.keys())\n\ndict_keys(['AccessKeyId', 'SecretAccessKey', 'SessionToken', 'Expiration'])\n\n\nCheck the expiration timestamp for the temporary credentials:\n\ncreds['Expiration']\n\n'2021-06-11T02:36:29.000Z'\n\n\nOpen zarr datasets with s3fs and xarray\nGet the s3 output directory and list of zarr datasets from the list of links. The s3 directory should be the fifth item; the urls are from item six onward:\n\ns3_dir = response['links'][4]['href']\n\nprint(s3_dir)\n\ns3://harmony-prod-staging/public/harmony/netcdf-to-zarr/2295236b-8086-4543-9482-f524a9f2d0c3/\n\n\nNow select the URLs for the staged files and print the first one:\n\ns3_urls = [u['href'] for u in response['links'][5:]]\n\nprint(s3_urls[0])\n\ns3://harmony-prod-staging/public/harmony/netcdf-to-zarr/2295236b-8086-4543-9482-f524a9f2d0c3/OCEAN_BOTTOM_PRESSURE_mon_mean_2009-12_ECCO_V4r4_latlon_0p50deg.zarr\n\n\nUse the AWS s3fs package and your temporary aws_creds to open the zarr directory storage:\n\ns3 = s3fs.S3FileSystem(\n key=creds['AccessKeyId'],\n secret=creds['SecretAccessKey'],\n token=creds['SessionToken'],\n client_kwargs={'region_name':'us-west-2'},\n)\n\nlen(s3.ls(s3_dir))\n\n97\n\n\nPlot the first Ocean Bottom Pressure dataset\nCheck out the documentation for xarray’s open_zarr method at this link. Open the first dataset and plot the OBP variable:\n\nds0 = xr.open_zarr(s3.get_mapper(s3_urls[0]), decode_cf=True, mask_and_scale=True)\n\n# Mask the dataset where OBP is not within the bounds of the variable's valid min/max:\nds0_masked = ds0.where((ds0.OBP>=ds0.OBP.valid_min) & (ds0.OBP<=ds0.OBP.valid_max))\n\n# Plot the masked dataset\nds0_masked.OBP.isel(time=0).plot.imshow(size=10)\n\n<matplotlib.image.AxesImage at 0x7f28ed2ba4c0>\n\n\n\n\n\nLoad the zarr datasets into one large xarray dataset\nLoad all the datasets in a loop and concatenate them:\n\nzds = xr.concat([xr.open_zarr(s3.get_mapper(u)) for u in s3_urls], dim=\"time\")\n\nprint(zds)\n\n<xarray.Dataset>\nDimensions: (latitude: 360, longitude: 720, nv: 2, time: 97)\nCoordinates:\n * latitude (latitude) float64 -89.75 -89.25 -88.75 ... 89.25 89.75\n latitude_bnds (latitude, nv) float64 -90.0 -89.5 -89.5 ... 89.5 89.5 90.0\n * longitude (longitude) float64 -179.8 -179.2 -178.8 ... 179.2 179.8\n longitude_bnds (longitude, nv) float64 -180.0 -179.5 -179.5 ... 179.5 180.0\n * time (time) datetime64[ns] 2009-12-16T12:00:00 ... 2017-12-16T...\n time_bnds (time, nv) datetime64[ns] dask.array<chunksize=(1, 2), meta=np.ndarray>\nDimensions without coordinates: nv\nData variables:\n OBP (time, latitude, longitude) float64 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\n OBPGMAP (time, latitude, longitude) float64 dask.array<chunksize=(1, 360, 720), meta=np.ndarray>\nAttributes: (12/57)\n Conventions: CF-1.8, ACDD-1.3\n acknowledgement: This research was carried out by the Jet Pr...\n author: Ian Fenty and Ou Wang\n cdm_data_type: Grid\n comment: Fields provided on a regular lat-lon grid. ...\n coordinates_comment: Note: the global 'coordinates' attribute de...\n ... ...\n time_coverage_duration: P1M\n time_coverage_end: 2010-01-01T00:00:00\n time_coverage_resolution: P1M\n time_coverage_start: 2009-12-01T00:00:00\n title: ECCO Ocean Bottom Pressure - Monthly Mean 0...\n uuid: 297c8df0-4158-11eb-b208-0cc47a3f687b\n\n\nReference OBP and mask the dataset according to the valid minimum and maximum:\n\nobp = zds.OBP\n\nprint(obp)\n\n<xarray.DataArray 'OBP' (time: 97, latitude: 360, longitude: 720)>\ndask.array<concatenate, shape=(97, 360, 720), dtype=float64, chunksize=(1, 360, 720), chunktype=numpy.ndarray>\nCoordinates:\n * latitude (latitude) float64 -89.75 -89.25 -88.75 ... 88.75 89.25 89.75\n * longitude (longitude) float64 -179.8 -179.2 -178.8 ... 178.8 179.2 179.8\n * time (time) datetime64[ns] 2009-12-16T12:00:00 ... 2017-12-16T06:00:00\nAttributes:\n comment: OBP excludes the contribution from global mean at...\n coverage_content_type: modelResult\n long_name: Ocean bottom pressure given as equivalent water t...\n units: m\n valid_max: 72.07011413574219\n valid_min: -1.7899188995361328\n\n\nGet the valid min and max from the corresponding CF attributes:\n\nobp_vmin, obp_vmax = obp.valid_min, obp.valid_max\n\nobp_vmin, obp_vmax\n\n(-1.7899188995361328, 72.07011413574219)\n\n\nMask the dataset according to the OBP min and max and plot a series:\n\n# Mask dataset where not inside OBP variable valid min/max:\nzds_masked = zds.where((obp>=obp_vmin)&(obp<=obp_vmax))\n\n# Plot SSH again for the first 12 time slices:\nobpp = zds_masked.OBP.isel(time=slice(0, 6)).plot(\n x=\"longitude\", \n y=\"latitude\", \n col=\"time\",\n levels=8,\n col_wrap=3, \n add_colorbar=False,\n figsize=(14, 8)\n)\n\n# Plot a colorbar on a secondary axis\nmappable = obpp.axes[0][0].collections[0]\ncax = plt.axes([0.05, -0.04, 0.95, 0.04])\ncbar1 = plt.colorbar(mappable, cax=cax, orientation='horizontal')" }, { - "objectID": "external/data_discovery_cmr-stac_api.html#split-data-links-list-into-logical-groupings", - "href": "external/data_discovery_cmr-stac_api.html#split-data-links-list-into-logical-groupings", - "title": "Data Discovery with CMR-STAC API", - "section": "Split Data Links List into Logical Groupings", - "text": "Split Data Links List into Logical Groupings\nWe have a list of links to data assets that meet our search and filtering criteria. Below we’ll split our list from above into lists first by tile/UTM zone and then further by individual bands bands. The commands that follow will do the splitting with python routines.\n\nSplit by UTM tile specified in the file name (e.g., T14TKL & T13TGF)\n\ntile_dicts = defaultdict(list) # https://stackoverflow.com/questions/26367812/appending-to-list-in-python-dictionary\n\n\nfor l in evi_band_links:\n tile = l.split('.')[-6]\n tile_dicts[tile].append(l)\n\n\nPrint dictionary keys and values, i.e. the data links\n\ntile_dicts.keys()\n\ndict_keys(['T13TGF', 'T14TKL'])\n\n\n\ntile_dicts['T13TGF'][:5]\n\n['https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B05.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.Fmask.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B02.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021133T173859.v2.0/HLS.S30.T13TGF.2021133T173859.v2.0.B8A.tif']\n\n\nNow we will create a separate list of data links for each tile\n\ntile_links_T14TKL = tile_dicts['T14TKL']\ntile_links_T13TGF = tile_dicts['T13TGF']\n\n\n\nPrint band/layer links for HLS tile T13TGF\n\n# tile_links_T13TGF[:10]\n\n\n\n\nSplit the links by band\n\nbands_dicts = defaultdict(list)\n\n\nfor b in tile_links_T13TGF:\n band = b.split('.')[-2]\n bands_dicts[band].append(b)\n\n\nbands_dicts.keys()\n\ndict_keys(['B04', 'B05', 'Fmask', 'B02', 'B8A'])\n\n\n\nbands_dicts['B04']\n\n['https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021133T172406.v2.0/HLS.L30.T13TGF.2021133T172406.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021133T173859.v2.0/HLS.S30.T13TGF.2021133T173859.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021140T173021.v2.0/HLS.L30.T13TGF.2021140T173021.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021140T172859.v2.0/HLS.S30.T13TGF.2021140T172859.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021145T172901.v2.0/HLS.S30.T13TGF.2021145T172901.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021155T172901.v2.0/HLS.S30.T13TGF.2021155T172901.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021156T173029.v2.0/HLS.L30.T13TGF.2021156T173029.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021158T173901.v2.0/HLS.S30.T13TGF.2021158T173901.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021163T173909.v2.0/HLS.S30.T13TGF.2021163T173909.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021165T172422.v2.0/HLS.L30.T13TGF.2021165T172422.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021165T172901.v2.0/HLS.S30.T13TGF.2021165T172901.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021173T173909.v2.0/HLS.S30.T13TGF.2021173T173909.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021185T172901.v2.0/HLS.S30.T13TGF.2021185T172901.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021188T173037.v2.0/HLS.L30.T13TGF.2021188T173037.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021190T172859.v2.0/HLS.S30.T13TGF.2021190T172859.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021193T173909.v2.0/HLS.S30.T13TGF.2021193T173909.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021198T173911.v2.0/HLS.S30.T13TGF.2021198T173911.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021200T172859.v2.0/HLS.S30.T13TGF.2021200T172859.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021203T173909.v2.0/HLS.S30.T13TGF.2021203T173909.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021204T173042.v2.0/HLS.L30.T13TGF.2021204T173042.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021208T173911.v2.0/HLS.S30.T13TGF.2021208T173911.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021210T172859.v2.0/HLS.S30.T13TGF.2021210T172859.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021215T172901.v2.0/HLS.S30.T13TGF.2021215T172901.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021218T173911.v2.0/HLS.S30.T13TGF.2021218T173911.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021220T173049.v2.0/HLS.L30.T13TGF.2021220T173049.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021220T172859.v2.0/HLS.S30.T13TGF.2021220T172859.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021223T173909.v2.0/HLS.S30.T13TGF.2021223T173909.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021228T173911.v2.0/HLS.S30.T13TGF.2021228T173911.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T13TGF.2021229T172441.v2.0/HLS.L30.T13TGF.2021229T172441.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021230T172859.v2.0/HLS.S30.T13TGF.2021230T172859.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021235T172901.v2.0/HLS.S30.T13TGF.2021235T172901.v2.0.B04.tif',\n 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TGF.2021243T173859.v2.0/HLS.S30.T13TGF.2021243T173859.v2.0.B04.tif']" + "objectID": "external/sentinel-6_opendap_access_gridding.html", + "href": "external/sentinel-6_opendap_access_gridding.html", + "title": "Sentinel-6 MF L2 Altimetry Data Access (OPeNDAP) & Gridding", + "section": "", + "text": "imported on: 2023-05-30\n\nThis notebook was originally developed for the 2021 Cloud Hackathon, and has been updated with most current approaches.\n\n\nThe original source for this document is https://nasa-openscapes.github.io/2021-Cloud-Hackathon/tutorials/06_S6_OPeNDAP_Access_Gridding.html\n\n\n06. Sentinel-6 MF L2 Altimetry Data Access (OPeNDAP) & Gridding\nIn this tutorial you will learn…\n\nabout level 2 radar altimetry data from the Sentinel-6 Michael Freilich mission;\nhow to efficiently download variable subsets using OPeNDAP;\nhow to grid the along-track altimetry observations produced by S6 at level 2.;\n\n\nAbout Ocean Surface Topography (OST)\n\nThe primary contribution of satellite altimetry to satellite oceanography has been to:\n\nImprove the knowledge of ocean tides and develop global tide models.\nMonitor the variation of global mean sea level and its relationship to changes in ocean mass and heat content.\nMap the general circulation variability of the ocean, including the ocean mesoscale, over decades and in near real-time using multi-satellite altimetric sampling.\n\n\n\nAbout Sentinel-6 MF\nhttps://search.earthdata.nasa.gov/search?fpj=Sentinel-6\n\nhttps://podaac.jpl.nasa.gov/Sentinel-6\n\nMission Characteristics\nSemi-major axis: 7,714.43 km\nEccentricity: 0.000095\nInclination: 66.04°\nArgument of periapsis: 90.0°\nMean anomaly: 253.13°\nReference altitude: 1,336 km\nNodal period: 6,745.72 sec\nRepeat period: 9.9156 days\nNumber of revolutions within a cycle: 127\nNumber of passes within a cycle: 254\nEquatorial cross track separation: 315 km\nGround track control band: +1 km\nAcute angle at Equator crossings: 39.5°\nGround track speed: 5.8 km/s\n\n\n\nRequirements\nThis workflow was developed using Python 3.9 (and tested against versions 3.7, 3.8).\n\nimport os\nimport tqdm\nimport numpy as np\nimport xarray as xr\nimport matplotlib.pyplot as plt\nfrom concurrent.futures import ThreadPoolExecutor\nfrom pyresample.kd_tree import resample_gauss\nimport pyresample as pr\n\n\n\nDataset\nhttps://podaac.jpl.nasa.gov/dataset/JASON_CS_S6A_L2_ALT_LR_RED_OST_NRT_F\nThis example operates on Level 2 Low Resolution Altimetry from Sentinel-6 Michael Freilich (the Near Real Time Reduced distribution). It is most easily identified by its collection ShortName, given below with the more cryptic concept-id, it’s unique identifier in the CMR.\n\nShortName = 'JASON_CS_S6A_L2_ALT_LR_RED_OST_NRT_F'\nconcept_id = 'C1968980576-POCLOUD'\n\n\n\ncycle = 25\n\nurl = f\"https://cmr.earthdata.nasa.gov/search/granules.csv?ShortName={ShortName}&cycle={cycle}&page_size=200\"\n\nprint(url)\n\nhttps://cmr.earthdata.nasa.gov/search/granules.csv?ShortName=JASON_CS_S6A_L2_ALT_LR_RED_OST_NRT_F&cycle=25&page_size=200\n\n\ncurl --silent --output\ncat\ntail --lines\ncut --delimiter --fields\n\n!curl --silent --output \"results.csv\" \"$url\"\n\nfiles = !cat results.csv | tail --lines=+2 | cut --delimiter=',' --fields=5 | cut --delimiter='/' --fields=6\n\nprint(files.s.replace(\" \", \"\\n\"))\n\nS6A_P4_2__LR_RED__NR_025_001_20210713T162644_20210713T182234_F02.nc\nS6A_P4_2__LR_RED__NR_025_003_20210713T182234_20210713T201839_F02.nc\nS6A_P4_2__LR_RED__NR_025_006_20210713T201839_20210713T215450_F02.nc\nS6A_P4_2__LR_RED__NR_025_007_20210713T215450_20210713T234732_F02.nc\nS6A_P4_2__LR_RED__NR_025_009_20210713T234732_20210714T014224_F02.nc\nS6A_P4_2__LR_RED__NR_025_011_20210714T014224_20210714T033812_F02.nc\nS6A_P4_2__LR_RED__NR_025_013_20210714T033812_20210714T053356_F02.nc\nS6A_P4_2__LR_RED__NR_025_015_20210714T053357_20210714T072934_F02.nc\nS6A_P4_2__LR_RED__NR_025_017_20210714T072934_20210714T090919_F02.nc\nS6A_P4_2__LR_RED__NR_025_019_20210714T090919_20210714T110146_F02.nc\nS6A_P4_2__LR_RED__NR_025_021_20210714T110146_20210714T125702_F02.nc\nS6A_P4_2__LR_RED__NR_025_023_20210714T125702_20210714T145316_F02.nc\nS6A_P4_2__LR_RED__NR_025_025_20210714T145317_20210714T164922_F02.nc\nS6A_P4_2__LR_RED__NR_025_027_20210714T164922_20210714T184510_F02.nc\nS6A_P4_2__LR_RED__NR_025_029_20210714T184510_20210714T204143_F02.nc\nS6A_P4_2__LR_RED__NR_025_032_20210714T204143_20210714T221611_F02.nc\nS6A_P4_2__LR_RED__NR_025_033_20210714T221611_20210715T000941_F02.nc\nS6A_P4_2__LR_RED__NR_025_035_20210715T000941_20210715T020456_F02.nc\nS6A_P4_2__LR_RED__NR_025_037_20210715T020456_20210715T040047_F02.nc\nS6A_P4_2__LR_RED__NR_025_039_20210715T040047_20210715T055630_F02.nc\nS6A_P4_2__LR_RED__NR_025_041_20210715T055630_20210715T075208_F02.nc\nS6A_P4_2__LR_RED__NR_025_043_20210715T075208_20210715T093037_F02.nc\nS6A_P4_2__LR_RED__NR_025_045_20210715T093037_20210715T112356_F02.nc\nS6A_P4_2__LR_RED__NR_025_047_20210715T112356_20210715T131944_F02.nc\nS6A_P4_2__LR_RED__NR_025_049_20210715T131944_20210715T151600_F02.nc\nS6A_P4_2__LR_RED__NR_025_051_20210715T151602_20210715T165851_F02.nc\nS6A_P4_2__LR_RED__NR_025_053_20210715T171228_20210715T190748_F02.nc\nS6A_P4_2__LR_RED__NR_025_056_20210715T190748_20210715T204627_F02.nc\nS6A_P4_2__LR_RED__NR_025_057_20210715T204627_20210715T223758_F02.nc\nS6A_P4_2__LR_RED__NR_025_059_20210715T223758_20210716T003159_F02.nc\nS6A_P4_2__LR_RED__NR_025_061_20210716T003159_20210716T022732_F02.nc\nS6A_P4_2__LR_RED__NR_025_063_20210716T022732_20210716T042333_F02.nc\nS6A_P4_2__LR_RED__NR_025_065_20210716T042333_20210716T061901_F02.nc\nS6A_P4_2__LR_RED__NR_025_067_20210716T061901_20210716T081446_F02.nc\nS6A_P4_2__LR_RED__NR_025_070_20210716T081446_20210716T095203_F02.nc\nS6A_P4_2__LR_RED__NR_025_071_20210716T095203_20210716T114624_F02.nc\nS6A_P4_2__LR_RED__NR_025_073_20210716T114624_20210716T134228_F02.nc\nS6A_P4_2__LR_RED__NR_025_075_20210716T134228_20210716T153841_F02.nc\nS6A_P4_2__LR_RED__NR_025_077_20210716T153841_20210716T173433_F02.nc\nS6A_P4_2__LR_RED__NR_025_079_20210716T173433_20210716T193033_F02.nc\nS6A_P4_2__LR_RED__NR_025_082_20210716T193033_20210716T210718_F02.nc\nS6A_P4_2__LR_RED__NR_025_083_20210716T210718_20210716T225942_F02.nc\nS6A_P4_2__LR_RED__NR_025_085_20210716T225942_20210717T005425_F02.nc\nS6A_P4_2__LR_RED__NR_025_087_20210717T005425_20210717T025012_F02.nc\nS6A_P4_2__LR_RED__NR_025_089_20210717T025012_20210717T044557_F02.nc\nS6A_P4_2__LR_RED__NR_025_091_20210717T044557_20210717T064133_F02.nc\nS6A_P4_2__LR_RED__NR_025_093_20210717T064133_20210717T082134_F02.nc\nS6A_P4_2__LR_RED__NR_025_095_20210717T082134_20210717T101352_F02.nc\nS6A_P4_2__LR_RED__NR_025_097_20210717T101352_20210717T120859_F02.nc\nS6A_P4_2__LR_RED__NR_025_099_20210717T120859_20210717T140513_F02.nc\nS6A_P4_2__LR_RED__NR_025_101_20210717T140513_20210717T160120_F02.nc\nS6A_P4_2__LR_RED__NR_025_103_20210717T160120_20210717T175708_F02.nc\nS6A_P4_2__LR_RED__NR_025_105_20210717T175708_20210717T195329_F02.nc\nS6A_P4_2__LR_RED__NR_025_108_20210717T195329_20210717T212832_F02.nc\nS6A_P4_2__LR_RED__NR_025_109_20210717T212832_20210717T232147_F02.nc\nS6A_P4_2__LR_RED__NR_025_111_20210717T232147_20210718T011655_F02.nc\nS6A_P4_2__LR_RED__NR_025_113_20210718T011655_20210718T031245_F02.nc\nS6A_P4_2__LR_RED__NR_025_115_20210718T031245_20210718T050829_F02.nc\nS6A_P4_2__LR_RED__NR_025_117_20210718T050829_20210718T070406_F02.nc\nS6A_P4_2__LR_RED__NR_025_119_20210718T070406_20210718T084306_F02.nc\nS6A_P4_2__LR_RED__NR_025_121_20210718T084306_20210718T103559_F02.nc\nS6A_P4_2__LR_RED__NR_025_123_20210718T103559_20210718T123140_F02.nc\nS6A_P4_2__LR_RED__NR_025_125_20210718T123140_20210718T142756_F02.nc\nS6A_P4_2__LR_RED__NR_025_127_20210718T142756_20210718T162356_F02.nc\nS6A_P4_2__LR_RED__NR_025_129_20210718T162356_20210718T181945_F02.nc\nS6A_P4_2__LR_RED__NR_025_132_20210718T181945_20210718T195907_F02.nc\nS6A_P4_2__LR_RED__NR_025_133_20210718T195907_20210718T215014_F02.nc\nS6A_P4_2__LR_RED__NR_025_135_20210718T215014_20210718T234402_F02.nc\nS6A_P4_2__LR_RED__NR_025_137_20210718T234402_20210719T013937_F02.nc\nS6A_P4_2__LR_RED__NR_025_139_20210719T013937_20210719T033531_F02.nc\nS6A_P4_2__LR_RED__NR_025_141_20210719T033531_20210719T053101_F02.nc\nS6A_P4_2__LR_RED__NR_025_143_20210719T053101_20210719T072643_F02.nc\nS6A_P4_2__LR_RED__NR_025_146_20210719T072643_20210719T090425_F02.nc\nS6A_P4_2__LR_RED__NR_025_147_20210719T090425_20210719T105824_F02.nc\nS6A_P4_2__LR_RED__NR_025_149_20210719T105824_20210719T125424_F02.nc\nS6A_P4_2__LR_RED__NR_025_151_20210719T125424_20210719T145038_F02.nc\nS6A_P4_2__LR_RED__NR_025_153_20210719T145541_20210719T164632_F02.nc\nS6A_P4_2__LR_RED__NR_025_155_20210719T164632_20210719T184227_F02.nc\nS6A_P4_2__LR_RED__NR_025_158_20210719T184227_20210719T201949_F02.nc\nS6A_P4_2__LR_RED__NR_025_159_20210719T201949_20210719T221154_F02.nc\nS6A_P4_2__LR_RED__NR_025_161_20210719T221154_20210720T000626_F02.nc\nS6A_P4_2__LR_RED__NR_025_163_20210720T000626_20210720T020212_F02.nc\nS6A_P4_2__LR_RED__NR_025_165_20210720T020212_20210720T035756_F02.nc\nS6A_P4_2__LR_RED__NR_025_167_20210720T035756_20210720T055333_F02.nc\nS6A_P4_2__LR_RED__NR_025_169_20210720T055333_20210720T073350_F02.nc\nS6A_P4_2__LR_RED__NR_025_171_20210720T073350_20210720T092602_F02.nc\nS6A_P4_2__LR_RED__NR_025_173_20210720T092602_20210720T112057_F02.nc\nS6A_P4_2__LR_RED__NR_025_175_20210720T112057_20210720T131708_F02.nc\nS6A_P4_2__LR_RED__NR_025_177_20210720T131708_20210720T151317_F02.nc\nS6A_P4_2__LR_RED__NR_025_184_20210720T190549_20210720T204056_F02.nc\nS6A_P4_2__LR_RED__NR_025_185_20210720T204056_20210720T223355_F02.nc\nS6A_P4_2__LR_RED__NR_025_187_20210720T223355_20210721T002855_F02.nc\nS6A_P4_2__LR_RED__NR_025_189_20210721T002855_20210721T022443_F02.nc\nS6A_P4_2__LR_RED__NR_025_191_20210721T022443_20210721T042029_F02.nc\nS6A_P4_2__LR_RED__NR_025_193_20210721T042029_20210721T061605_F02.nc\nS6A_P4_2__LR_RED__NR_025_195_20210721T061605_20210721T075531_F02.nc\nS6A_P4_2__LR_RED__NR_025_197_20210721T075531_20210721T094805_F02.nc\nS6A_P4_2__LR_RED__NR_025_199_20210721T094805_20210721T114336_F02.nc\nS6A_P4_2__LR_RED__NR_025_201_20210721T114336_20210721T133952_F02.nc\nS6A_P4_2__LR_RED__NR_025_203_20210721T133952_20210721T153555_F02.nc\nS6A_P4_2__LR_RED__NR_025_205_20210721T153555_20210721T173143_F02.nc\nS6A_P4_2__LR_RED__NR_025_207_20210721T173143_20210721T191151_F02.nc\nS6A_P4_2__LR_RED__NR_025_209_20210721T191151_20210721T210223_F02.nc\nS6A_P4_2__LR_RED__NR_025_211_20210721T210223_20210721T225607_F02.nc\nS6A_P4_2__LR_RED__NR_025_213_20210721T225607_20210722T005131_F02.nc\nS6A_P4_2__LR_RED__NR_025_215_20210722T005131_20210722T024724_F02.nc\nS6A_P4_2__LR_RED__NR_025_217_20210722T024724_20210722T044301_F02.nc\nS6A_P4_2__LR_RED__NR_025_219_20210722T044301_20210722T063841_F02.nc\nS6A_P4_2__LR_RED__NR_025_221_20210722T063841_20210722T081646_F02.nc\nS6A_P4_2__LR_RED__NR_025_223_20210722T081646_20210722T101025_F02.nc\nS6A_P4_2__LR_RED__NR_025_225_20210722T101025_20210722T120619_F02.nc\nS6A_P4_2__LR_RED__NR_025_227_20210722T120619_20210722T140235_F02.nc\nS6A_P4_2__LR_RED__NR_025_229_20210722T140235_20210722T155831_F02.nc\nS6A_P4_2__LR_RED__NR_025_231_20210722T155831_20210722T175423_F02.nc\nS6A_P4_2__LR_RED__NR_025_234_20210722T175423_20210722T193222_F02.nc\nS6A_P4_2__LR_RED__NR_025_235_20210722T193222_20210722T212406_F02.nc\nS6A_P4_2__LR_RED__NR_025_237_20210722T212406_20210722T231828_F02.nc\nS6A_P4_2__LR_RED__NR_025_239_20210722T231828_20210723T011405_F02.nc\nS6A_P4_2__LR_RED__NR_025_241_20210723T011405_20210723T030955_F02.nc\nS6A_P4_2__LR_RED__NR_025_243_20210723T030955_20210723T050533_F02.nc\nS6A_P4_2__LR_RED__NR_025_245_20210723T050533_20210723T064603_F02.nc\nS6A_P4_2__LR_RED__NR_025_247_20210723T064603_20210723T083817_F02.nc\nS6A_P4_2__LR_RED__NR_025_249_20210723T083817_20210723T103256_F02.nc\nS6A_P4_2__LR_RED__NR_025_251_20210723T103256_20210723T122904_F02.nc\nS6A_P4_2__LR_RED__NR_025_253_20210723T122904_20210723T142514_F02.nc\n\n\n\nOPeNDAP\nhttps://opendap.github.io/documentation/UserGuideComprehensive.html#Constraint_Expressions (Hyrax/OPeNDAP docs)\n\ntmp = files.l[0].split('.')[0]\n\nprint(f\"https://opendap.earthdata.nasa.gov/collections/{concept_id}/granules/{tmp}.html\")\n\nhttps://opendap.earthdata.nasa.gov/collections/C1968980576-POCLOUD/granules/S6A_P4_2__LR_RED__NR_025_001_20210713T162644_20210713T182234_F02.html\n\n\n\nvariables = ['data_01_time',\n 'data_01_longitude',\n 'data_01_latitude',\n 'data_01_ku_ssha']\n\n\nv = \",\".join(variables)\n\nurls = []\nfor f in files:\n urls.append(f\"https://opendap.earthdata.nasa.gov/collections/{concept_id}/granules/{f}4?{v}\")\n\nprint(urls[0])\n\nhttps://opendap.earthdata.nasa.gov/collections/C1968980576-POCLOUD/granules/S6A_P4_2__LR_RED__NR_025_001_20210713T162644_20210713T182234_F02.nc4?data_01_time,data_01_longitude,data_01_latitude,data_01_ku_ssha\n\n\n\n\n\nDownload Subsets\nThese functions download one granule from the remote source to a local target, and will reliably manage simultaneous streaming downloads divided between multiple threads.\n\nwith python3:\nimport requests\n\ndef download(source: str, target: str):\n with requests.get(source, stream=True) as remote, open(target, 'wb') as local:\n if remote.status_code // 100 == 2: \n for chunk in remote.iter_content(chunk_size=1024):\n if chunk:\n local.write(chunk)\n\n\nwith wget:\n\ndef download(source: str):\n \n target = os.path.basename(source.split(\"?\")[0])\n \n if not os.path.isfile(target):\n !wget --quiet --continue --output-document $target $source\n \n return target\n\n\nn_workers = 12\n\nwith ThreadPoolExecutor(max_workers=n_workers) as pool:\n\n workers = pool.map(download, urls)\n \n files = list(tqdm.tqdm(workers, total=len(urls)))\n\n100%|██████████| 125/125 [01:01<00:00, 2.05it/s]\n\n\nhttps://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor\nThe source files range from 2.5MB to 3.0MB. These OPeNDAP subsets are ~100KB apiece. (anecdote: it took less than 10 minutes to download subsets for >1700 granules/files when I ran this routine for all cycles going back to 2021-06-22.)\n\n!du -sh .\n\n17M .\n\n\nhttps://www.gnu.org/software/coreutils/manual/html_node/du-invocation.html\n\n\nAggregate cycle\nSort the list of local subsets to ensure they concatenate in proper order. Call open_mfdataset on the list to open all the subsets in memory as one dataset in xarray.\n\nds = xr.open_mfdataset(sorted(files))\n\nprint(ds)\n\n<xarray.Dataset>\nDimensions: (data_01_time: 827001)\nCoordinates:\n * data_01_time (data_01_time) datetime64[ns] 2021-07-13T16:26:45 ... ...\nData variables:\n data_01_longitude (data_01_time) float64 dask.array<chunksize=(6950,), meta=np.ndarray>\n data_01_latitude (data_01_time) float64 dask.array<chunksize=(6950,), meta=np.ndarray>\n data_01_ku_ssha (data_01_time) float64 dask.array<chunksize=(6950,), meta=np.ndarray>\nAttributes: (12/63)\n Convention: CF-1.7\n institution: EUMETSAT\n references: Sentinel-6_Jason-CS ALT Generic P...\n contact: ops@eumetsat.int\n radiometer_sensor_name: AMR-C\n doris_sensor_name: DORIS\n ... ...\n xref_solid_earth_tide: S6__P4_2__SETD_AX_20151008T000000...\n xref_surface_classification: S6__P4____SURF_AX_20151008T000000...\n xref_wind_speed_alt: S6A_P4_2__WNDL_AX_20151008T000000...\n product_name: S6A_P4_2__LR______20210713T162644...\n history: 2021-07-13 18:38:07 : Creation\\n2...\n history_json: [{\"$schema\":\"https:\\/\\/harmony.ea...\n\n\nhttps://xarray.pydata.org/en/stable/generated/xarray.open_mfdataset.html\nMake a dictionary to rename variables so that the data_01_ prefix is removed from each one.\n\nnew_variable_names = list(map(lambda x: x.split(\"_\")[-1], variables))\n\nmap_variable_names = dict(zip(variables, new_variable_names))\n\nmap_variable_names\n\n{'data_01_time': 'time',\n 'data_01_longitude': 'longitude',\n 'data_01_latitude': 'latitude',\n 'data_01_ku_ssha': 'ssha'}\n\n\nhttps://docs.python.org/3/library/functions.html#map\nhttps://docs.python.org/3/library/functions.html#zip\n\nds = ds.rename(map_variable_names)\n\nprint(list(ds.variables))\n\n['longitude', 'latitude', 'ssha', 'time']\n\n\n\n\nPlot ssha variable\nhttps://xarray.pydata.org/en/stable/generated/xarray.Dataset.rename.html\n\nds.plot.scatter( y=\"latitude\",\n x=\"longitude\", \n hue=\"ssha\",\n s=1,\n vmin=-0.4,\n vmax=0.4,\n levels=9, \n cmap=\"jet\",\n aspect=2.5,\n size=9, )\n\nplt.title(f\"ssha from s6 cycle {cycle}\")\nplt.xlim( 0., 360.)\nplt.ylim(-67., 67.)\nplt.show()\n\n\n\n\n\n\n\nBorrow 0.5-Degree Grid and Mask from ECCO V4r4\n\nAcknowledgement: This approach using pyresample was shared to me by Ian Fenty, ECCO Lead.\n\nhttps://search.earthdata.nasa.gov/search/granules?p=C2013583732-POCLOUD\nECCO V4r4 products are distributed in two spatial formats. One set of collections provides the ocean state estimates on the native model grid (LLC0090) and the other provides them after interpolating to a regular grid defined in geographic coordinates with horizontal cell size of 0.5-degrees.\nIt’s distributed as its own dataset/collection containing just one file. We can simply download it from the HTTPS download endpoint – the file size is inconsequential. The next cell downloads the file into the data folder from the granule’s https endpoint.\n\necco_url = \"https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/ECCO_L4_GEOMETRY_05DEG_V4R4/GRID_GEOMETRY_ECCO_V4r4_latlon_0p50deg.nc\"\n\necco_file = download(ecco_url)\n\necco_grid = xr.open_dataset(ecco_file)\n\nprint(ecco_grid)\n\n<xarray.Dataset>\nDimensions: (Z: 50, latitude: 360, longitude: 720, nv: 2)\nCoordinates:\n * Z (Z) float32 -5.0 -15.0 -25.0 ... -5.461e+03 -5.906e+03\n * latitude (latitude) float32 -89.75 -89.25 -88.75 ... 89.25 89.75\n * longitude (longitude) float32 -179.8 -179.2 -178.8 ... 179.2 179.8\n latitude_bnds (latitude, nv) float32 ...\n longitude_bnds (longitude, nv) float32 ...\n Z_bnds (Z, nv) float32 ...\nDimensions without coordinates: nv\nData variables:\n hFacC (Z, latitude, longitude) float64 ...\n Depth (latitude, longitude) float64 ...\n area (latitude, longitude) float64 ...\n drF (Z) float32 ...\n maskC (Z, latitude, longitude) bool ...\nAttributes: (12/57)\n acknowledgement: This research was carried out by the Jet...\n author: Ian Fenty and Ou Wang\n cdm_data_type: Grid\n comment: Fields provided on a regular lat-lon gri...\n Conventions: CF-1.8, ACDD-1.3\n coordinates_comment: Note: the global 'coordinates' attribute...\n ... ...\n references: ECCO Consortium, Fukumori, I., Wang, O.,...\n source: The ECCO V4r4 state estimate was produce...\n standard_name_vocabulary: NetCDF Climate and Forecast (CF) Metadat...\n summary: This dataset provides geometric paramete...\n title: ECCO Geometry Parameters for the 0.5 deg...\n uuid: b4795c62-86e5-11eb-9c5f-f8f21e2ee3e0\n\n\nhttps://xarray.pydata.org/en/stable/generated/xarray.open_dataset.html\nSelect index 0 on the Z axis/dimension to get the depth layer at ocean surface.\n\necco_grid = ecco_grid.isel(Z=0).copy()\n\nhttps://xarray.pydata.org/en/stable/generated/xarray.DataArray.isel.html\nThe maskC variable contains a boolean mask representing the wet/dry state of the area contained in each cell of the 3d grid defined by Z and latitude and longitude. Here are the variable’s attributes:\n\nprint(ecco_grid.maskC)\n\n<xarray.DataArray 'maskC' (latitude: 360, longitude: 720)>\n[259200 values with dtype=bool]\nCoordinates:\n Z float32 -5.0\n * latitude (latitude) float32 -89.75 -89.25 -88.75 ... 88.75 89.25 89.75\n * longitude (longitude) float32 -179.8 -179.2 -178.8 ... 178.8 179.2 179.8\nAttributes:\n coverage_content_type: modelResult\n long_name: wet/dry boolean mask for grid cell\n comment: True for grid cells with nonzero open vertical fr...\n\n\nPlot the land/water mask maskC:\n\necco_grid.maskC.plot()\n\n<matplotlib.collections.QuadMesh at 0x7fed730b00a0>\n\n\n\n\n\nhttps://xarray.pydata.org/en/stable/generated/xarray.DataArray.plot.html\nDefine target grid based on the longitudes and latitudes from the ECCO grid geometry dataset. This time define the grid using two 2-dimensional arrays that give positions of all SSHA values in geographic/longitude-latitude coordinates.\n\necco_lons = ecco_grid.maskC.longitude.values\necco_lats = ecco_grid.maskC.latitude.values\n\necco_lons_2d, ecco_lats_2d = np.meshgrid(ecco_lons, ecco_lats)\n\nprint(ecco_lons_2d.shape, ecco_lats_2d.shape)\n\n(360, 720) (360, 720)\n\n\nCreate the target swath definition from the 2d arrays of lons and lats from ECCO V4r4 0.5-degree grid.\n\ntgt = pr.SwathDefinition(ecco_lons_2d, ecco_lats_2d)\n\npyresample.geometry.SwathDefinition\n\n\nGrid ssha or other variable\nGet one timestamp to represent the midpoint of the 10-day cycle.\n\ntime = np.datetime64(ds['time'].mean().data)\n\nprint(time)\n\n2021-07-18T15:11:35.073955170\n\n\nAccess the target variable, ssha in this case. Make a nan mask from the ssha variable.\n\nnans = ~np.isnan(ds.ssha.values)\n\nssha = ds.ssha.values[nans]\n\nssha.shape\n\n(518164,)\n\n\nCreate the source swath definition from the 1d arrays of lons and lats from the S6 level-2 along-track altimetry time series.\n\nlons = ds.longitude.values[nans]\nlats = ds.latitude.values[nans]\n \nprint(lons.shape, lats.shape)\n\n(518164,) (518164,)\n\n\n\nlons = (lons + 180) % 360 - 180\n\nsrc = pr.SwathDefinition(lons, lats)\n\npyresample.geometry.SwathDefinition\nResample ssha data using kd-tree gaussian weighting neighbour approach.\n\nresult, stddev, counts = resample_gauss(\n src,\n ssha,\n tgt,\n radius_of_influence=175000, \n sigmas=25000,\n neighbours=100,\n fill_value=np.nan,\n with_uncert=True,\n)\n\nresult.shape\n\n/srv/conda/envs/notebook/lib/python3.9/site-packages/pyresample/kd_tree.py:384: UserWarning: Possible more than 100 neighbours within 175000 m for some data points\n warnings.warn(('Possible more than %s neighbours '\n\n\n(360, 720)\n\n\npyresample.kd_tree.resample_gauss\n\ndef to_xrda(data):\n return xr.DataArray(data,\n dims=['latitude', 'longitude'], \n coords={'time': time, \n 'latitude': ecco_lats, \n 'longitude': ecco_lons})\n\nApply the land/water mask in the numpy array created from the ECCO layer in the steps above. Then, convert the masked numpy array to an xarray data array object named gridded. Print its header.\n\ngrid = to_xrda(result)\n\ngrid.sel(latitude=slice(-67.0, 67.0)).plot(vmin=-0.4, vmax=0.4, cmap=\"jet\", figsize=(18, 7))\n\n<matplotlib.collections.QuadMesh at 0x7fed72fc43a0>\n\n\n\n\n\n\nstddev = to_xrda(stddev)\n\nstddev.sel(latitude=slice(-67.0, 67.0)).plot(robust=True, cmap=\"jet\", figsize=(18, 7))\n\n<matplotlib.collections.QuadMesh at 0x7fed72892520>\n\n\n\n\n\n\ncounts = to_xrda(counts)\n\ncounts.sel(latitude=slice(-67.0, 67.0)).plot(robust=True, cmap=\"jet\", figsize=(18, 7))\n\n<matplotlib.collections.QuadMesh at 0x7fed732a3af0>\n\n\n\n\n\n\n\nExercise\nCalculate area-weighted mean sea level.\n\n\nReferences\nnumpy (https://numpy.org/doc/stable/reference)\n\nnumpy.ndarray.data\n\nnumpy.where\n\nnumpy.isnan\n\ndatetimes\n\nnumpy.sum\n\nnumpy.nansum\n\nxarray (https://xarray.pydata.org/en/stable)\n\nxarray.DataArray\n\nxarray.DataArray.values\n\nxarray.DataArray.mean" }, { - "objectID": "external/data_discovery_cmr-stac_api.html#save-links-to-a-text-file", - "href": "external/data_discovery_cmr-stac_api.html#save-links-to-a-text-file", - "title": "Data Discovery with CMR-STAC API", - "section": "Save links to a text file", - "text": "Save links to a text file\nTo complete this exercise, we will save the individual link lists as separate text files with descriptive names.\nNOTE: If you are running the notebook from the tutorials-templates directory, please use the following path to write to the data directory: “../tutorials/data/{name}”\n\nWrite links from CMR-STAC API to a file\n\nfor k, v in bands_dicts.items():\n name = (f'HTTPS_T13TGF_{k}_Links.txt')\n with open(f'./data/{name}', 'w') as f: # use ../tutorials/data/{name} as your path if running the notebook from \"tutorials-template\"\n for l in v:\n f.write(f\"{l}\" + '\\n')\n\n\n\nWrite links to file for S3 access\n\nfor k, v in bands_dicts.items():\n name = (f'S3_T13TGF_{k}_Links.txt')\n with open(f'./data/{name}', 'w') as f: # use ../tutorials/data/{name} as your path if running the notebook from \"tutorials-template\"\n for l in v:\n s3l = l.replace('https://data.lpdaac.earthdatacloud.nasa.gov/', 's3://')\n f.write(f\"{s3l}\" + '\\n')" + "objectID": "external/harmony_subsetting.html", + "href": "external/harmony_subsetting.html", + "title": "Data Subsetting and Transformation Services in the Cloud", + "section": "", + "text": "imported on: 2023-05-30\nThis notebook was originally developed for the 2021 Cloud Hackathon, and has been updated with most current approaches." }, { - "objectID": "external/data_discovery_cmr-stac_api.html#resources", - "href": "external/data_discovery_cmr-stac_api.html#resources", - "title": "Data Discovery with CMR-STAC API", - "section": "Resources", - "text": "Resources\n\nSTAC Specification Webpage\nSTAC API Documentation\nCMR-STAC API Github\nPySTAC Client Documentation\nhttps://stackoverflow.com/questions/26367812/appending-to-list-in-python-dictionary\nGeopandas\nHLS Overview" + "objectID": "external/harmony_subsetting.html#using-the-harmony-py-library-to-access-customized-data-from-nasa-earthdata", + "href": "external/harmony_subsetting.html#using-the-harmony-py-library-to-access-customized-data-from-nasa-earthdata", + "title": "Data Subsetting and Transformation Services in the Cloud", + "section": "Using the Harmony-Py library to access customized data from NASA Earthdata", + "text": "Using the Harmony-Py library to access customized data from NASA Earthdata\n\n\nTiming:\n\nExercise: 40 minutes" }, { - "objectID": "external/xarray.html", - "href": "external/xarray.html", - "title": "Introduction to xarray", - "section": "", - "text": "imported on: 2023-05-30\nThis notebook was originally developed for the 2021 Cloud Hackathon, and has been updated with most current approaches." + "objectID": "external/harmony_subsetting.html#summary", + "href": "external/harmony_subsetting.html#summary", + "title": "Data Subsetting and Transformation Services in the Cloud", + "section": "Summary", + "text": "Summary\nWe have already explored direct access to the NASA EOSDIS archive in the cloud via the Amazon Simple Storage Service (S3) by using the Common Metadata Repository (CMR) to search for granule locations. In addition to directly accessing the files archived and distributed by each of the NASA DAACs, many datasets also support services that allow us to customize the data via subsetting, reformatting, reprojection, and other transformations.\nThis tutorial demonstrates how to find, request, and use customized data from a new ecosystem of services operating within the NASA Earthdata Cloud: NASA Harmony.\n\nBenefits\nBut first, why use this option when we’ve already learned how to access data directly from the NASA Earthdata Cloud?\n\nConsistent access patterns to EOSDIS holdings make cross-data center data access easier\nData reduction services allow us to request only the data we want, in the format and projection we want\nAnalysis Ready Data and cloud access will help reduce time-to-science\nCommunity Development helps reduce the barriers for re-use of code and sharing of domain knowledge\n\n\n\n\nData file filtering and subsetting\n\n\nSee more on the Earthdata Harmony landing page, including documentation on the Harmony API itself.\n\n\nObjectives\n\nConceptualize the data transformation service types and offerings provided by NASA Earthdata, including Harmony.\nPractice skills learned from the introductory CMR tutorial to discover what access and service options exist for a given data set, as well as variable metadata.\nUtilize the Harmony-py library to request subsetted MODIS L2 Sea Surface Temperature data over the Gulf of Mexico.\nRead Harmony subsetted outputs directly into xarray. ___" }, { - "objectID": "external/xarray.html#why-do-we-need-xarray", - "href": "external/xarray.html#why-do-we-need-xarray", - "title": "Introduction to xarray", - "section": "Why do we need xarray?", - "text": "Why do we need xarray?\nAs Geoscientists, we often work with time series of data with two or more dimensions: a time series of calibrated, orthorectified satellite images; two-dimensional grids of surface air temperature from an atmospheric reanalysis; or three-dimensional (level, x, y) cubes of ocean salinity from an ocean model. These data are often provided in GeoTIFF, NetCDF or HDF format with rich and useful metadata that we want to retain, or even use in our analysis. Common analyses include calculating means, standard deviations and anomalies over time or one or more spatial dimensions (e.g. zonal means). Model output often includes multiple variables that you want to apply similar analyses to.\n\n\n\nA schematic of multi-dimensional data\n\n\nThe schematic above shows a typical data structure for multi-dimensional data. There are two data cubes, one for temperature and one for precipitation. Common coordinate variables, in this case latitude, longitude and time are associated with each variable. Each variable, including coordinate variables, will have a set of attributes: name, units, missing value, etc. The file containing the data may also have attributes: source of the data, model name coordinate reference system if the data are projected. Writing code using low-level packages such as netcdf4 and numpy to read the data, then perform analysis, and write the results to file is time consuming and prone to errors." + "objectID": "external/harmony_subsetting.html#import-packages", + "href": "external/harmony_subsetting.html#import-packages", + "title": "Data Subsetting and Transformation Services in the Cloud", + "section": "Import Packages", + "text": "Import Packages\n\nfrom harmony import BBox, Client, Collection, Request, LinkType\nfrom harmony.config import Environment\nimport requests\nfrom pprint import pprint\nimport datetime as dt\nimport s3fs\nimport xarray as xr" }, { - "objectID": "external/xarray.html#what-is-xarray", - "href": "external/xarray.html#what-is-xarray", - "title": "Introduction to xarray", - "section": "What is xarray", - "text": "What is xarray\nxarray is an open-source project and python package to work with labelled multi-dimensional arrays. It is leverages numpy, pandas, matplotlib and dask to build Dataset and DataArray objects with built-in methods to subset, analyze, interpolate, and plot multi-dimensional data. It makes working with multi-dimensional data cubes efficient and fun. It will change your life for the better. You’ll be more attractive, more interesting, and better equiped to take on lifes challenges." + "objectID": "external/harmony_subsetting.html#discover-service-options-for-a-given-data-set", + "href": "external/harmony_subsetting.html#discover-service-options-for-a-given-data-set", + "title": "Data Subsetting and Transformation Services in the Cloud", + "section": "Discover service options for a given data set", + "text": "Discover service options for a given data set\n\nFirst, what do we mean by a “service”?\nIn the context of NASA Earthdata, we are usually referring to a service as any data transformation or customization process that packages or delivers data in a way that makes it easier to work with compared to how the data are natively archived at NASA EOSDIS. Basic customization options may include: * Subsetting (cropping) the data by: * Variable * Spatial boundary, * Temporal range * Reformatting * For example: From NetCDF-4 to Cloud Optimized GeoTIFF * Reprojection and/or Resampling * For example: From Sinusoidal to Polar Stereographic * Mosaicking * Aggregating\nA few main types or pathways for services that are commonly supported across the NASA DAACs include: * NASA Global Imagery Browse Service * Web services providing imagery, much of which is updated daily, to broaden accessibility of NASA EOSDIS data to the media and public. * Web Map Tile Service (WMTS) * Tiled Web Map Service (TWMS) * Web Map Service (WMS) * Keyhole Markup Language (KML) * Geospatial Data Abstraction Library (GDAL) * OPeNDAP * The Open-source Project for a Network Data Access Protocol is a NASA community standard DAP that provides a simple way to access and work with data over the internet. OPeNDAP’s client/server software allows us to subset and reformat data using an internet browser, command line interface, and other applications. * Harmony * In the most basic sense, Harmony is an Application Programming Interface, or API, allowing us to request customization options described above, which are then processed and returned as file outputs. Harmony helps to reduce pre-processing steps so we can spend less time preparing the data, and more time doing science.\nNote: These service offerings are unique to each NASA EOSDIS dataset.\nWhy is this?\nDue to varying levels of service, cloud migration status, and unique characteristics of the datasets themselves, not all service options are provided for all datasets. Therefore it is important to first explore a given dataset’s metadata to discover what service options are provided.\nLet’s utilize the CMR API skills we learned on Day 1 to inspect service metadata:\n\nurl = 'https://cmr.earthdata.nasa.gov/search'\n\nWe want to search by collection to inspect the access and service options that exist:\n\ncollection_url = f'{url}/{\"collections\"}'\n\nIn the CMR introduction tutorial, we explored cloud-hosted collections from different DAAC providers, and identified the CMR concept-id for a given data set id (also referred to as a short_name).\nHere we are jumping ahead and already know the concept_id we are interested in, by browsing cloud-hosted datasets from PO.DAAC in Earthdata Search: https://search.earthdata.nasa.gov/portal/podaac-cloud/search.\nWe are going to focus on MODIS_A-JPL-L2P-v2019.0: GHRSST Level 2P Global Sea Surface Skin Temperature from the Moderate Resolution Imaging Spectroradiometer (MODIS) on the NASA Aqua satellite (GDS2). Let’s first save this as a variable that we can use later on once we request data from Harmony.\n\nshort_name= 'MODIS_A-JPL-L2P-v2019.0'\nconcept_id = 'C1940473819-POCLOUD'\n\nWe will view the top-level metadata for this collection to see what additional service and variable metadata exist.\n\nresponse = requests.get(collection_url, \n params={\n 'concept_id': concept_id,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\nresponse = response.json()\n\nPrint the response:\n\npprint(response)\n\n{'feed': {'entry': [{'archive_center': 'NASA/JPL/PODAAC',\n 'associations': {'services': ['S1962070864-POCLOUD',\n 'S2004184019-POCLOUD'],\n 'tools': ['TL2108419875-POCLOUD',\n 'TL2092786348-POCLOUD'],\n 'variables': ['V1997812737-POCLOUD',\n 'V1997812697-POCLOUD',\n 'V2112014688-POCLOUD',\n 'V1997812756-POCLOUD',\n 'V1997812688-POCLOUD',\n 'V1997812670-POCLOUD',\n 'V1997812724-POCLOUD',\n 'V2112014684-POCLOUD',\n 'V1997812701-POCLOUD',\n 'V1997812681-POCLOUD',\n 'V2112014686-POCLOUD',\n 'V1997812663-POCLOUD',\n 'V1997812676-POCLOUD',\n 'V1997812744-POCLOUD',\n 'V1997812714-POCLOUD']},\n 'boxes': ['-90 -180 90 180'],\n 'browse_flag': True,\n 'collection_data_type': 'SCIENCE_QUALITY',\n 'coordinate_system': 'CARTESIAN',\n 'data_center': 'POCLOUD',\n 'dataset_id': 'GHRSST Level 2P Global Sea Surface Skin '\n 'Temperature from the Moderate Resolution '\n 'Imaging Spectroradiometer (MODIS) on the '\n 'NASA Aqua satellite (GDS2)',\n 'has_formats': True,\n 'has_spatial_subsetting': True,\n 'has_temporal_subsetting': True,\n 'has_transforms': False,\n 'has_variables': True,\n 'id': 'C1940473819-POCLOUD',\n 'links': [{'href': 'https://podaac.jpl.nasa.gov/Podaac/thumbnails/MODIS_A-JPL-L2P-v2019.0.jpg',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/browse#'},\n {'href': 'https://github.com/podaac/data-readers',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://podaac-tools.jpl.nasa.gov/drive/files/OceanTemperature/ghrsst/docs/GDS20r5.pdf',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://ghrsst.jpl.nasa.gov',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://oceancolor.gsfc.nasa.gov/atbd/sst/flag/',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://oceancolor.gsfc.nasa.gov/reprocessing/r2019/sst/',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://oceancolor.gsfc.nasa.gov/atbd/sst4/',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://modis.gsfc.nasa.gov/data/atbd/atbd_mod25.pdf',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://oceancolor.gsfc.nasa.gov/atbd/sst/',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'http://www.ghrsst.org',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://podaac.jpl.nasa.gov/forum/viewforum.php?f=18&sid=e2d67e5a01815fc6e39fcd2087ed8bc8',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://podaac.jpl.nasa.gov/CitingPODAAC',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://cmr.earthdata.nasa.gov/virtual-directory/collections/C1940473819-POCLOUD',\n 'hreflang': 'en-US',\n 'length': '75.0MB',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#'},\n {'href': 'https://github.com/podaac/tutorials/blob/master/notebooks/MODIS_L2P_SST_DataCube.ipynb',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#'},\n {'href': 'https://search.earthdata.nasa.gov/search/granules?p=C1940473819-POCLOUD',\n 'hreflang': 'en-US',\n 'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#'}],\n 'online_access_flag': True,\n 'orbit_parameters': {'inclination_angle': '98.1',\n 'number_of_orbits': '1.0',\n 'period': '98.4',\n 'swath_width': '2330.0'},\n 'organizations': ['NASA/JPL/PODAAC'],\n 'original_format': 'UMM_JSON',\n 'platforms': ['Aqua'],\n 'processing_level_id': '2',\n 'service_features': {'esi': {'has_formats': False,\n 'has_spatial_subsetting': False,\n 'has_temporal_subsetting': False,\n 'has_transforms': False,\n 'has_variables': False},\n 'harmony': {'has_formats': True,\n 'has_spatial_subsetting': True,\n 'has_temporal_subsetting': True,\n 'has_transforms': False,\n 'has_variables': True},\n 'opendap': {'has_formats': True,\n 'has_spatial_subsetting': True,\n 'has_temporal_subsetting': True,\n 'has_transforms': False,\n 'has_variables': True}},\n 'short_name': 'MODIS_A-JPL-L2P-v2019.0',\n 'summary': 'NASA produces skin sea surface temperature '\n '(SST) products from the Infrared (IR) '\n 'channels of the Moderate-resolution Imaging '\n 'Spectroradiometer (MODIS) onboard the Aqua '\n 'satellite. Aqua was launched by NASA on May '\n '4, 2002, into a sun synchronous, polar orbit '\n 'with a daylight ascending node at 1:30 pm, '\n 'formation flying in the A-train with other '\n 'Earth Observation Satellites (EOS), to study '\n 'the global dynamics of the Earth atmosphere, '\n 'land and oceans. MODIS captures data in 36 '\n 'spectral bands at a variety of spatial '\n 'resolutions. Two SST products can be present '\n 'in these files. The first is a skin SST '\n 'produced for both day and night (NSST) '\n 'observations, derived from the long wave IR '\n '11 and 12 micron wavelength channels, using a '\n 'modified nonlinear SST algorithm intended to '\n 'provide continuity of SST derived from '\n 'heritage and current NASA sensors. At night, '\n 'a second SST product is generated using the '\n 'mid-infrared 3.95 and 4.05 micron wavelength '\n 'channels which are unique to MODIS; the SST '\n 'derived from these measurements is identified '\n 'as SST4. The SST4 product has lower '\n 'uncertainty, but due to sun glint can only be '\n 'used at night. MODIS L2P SST data have a 1 km '\n 'spatial resolution at nadir and are stored in '\n '288 five minute granules per day. Full global '\n 'coverage is obtained every two days, with '\n 'coverage poleward of 32.3 degree being '\n 'complete each day. The production of MODIS '\n 'L2P SST files is part of the Group for High '\n 'Resolution Sea Surface Temperature (GHRSST) '\n 'project and is a joint collaboration between '\n 'the NASA Jet Propulsion Laboratory (JPL), the '\n 'NASA Ocean Biology Processing Group (OBPG), '\n 'and the Rosenstiel School of Marine and '\n 'Atmospheric Science (RSMAS). Researchers at '\n 'RSMAS are responsible for SST algorithm '\n 'development, error statistics and quality '\n 'flagging, while the OBPG, as the NASA ground '\n 'data system, is responsible for the '\n 'production of daily MODIS ocean products. JPL '\n 'acquires MODIS ocean granules from the OBPG '\n 'and reformats them to the GHRSST L2P netCDF '\n 'specification with complete metadata and '\n 'ancillary variables, and distributes the data '\n 'as the official Physical Oceanography Data '\n 'Archive (PO.DAAC) for SST. The R2019.0 '\n 'supersedes the previous R2014.0 datasets '\n 'which can be found at '\n 'https://doi.org/10.5067/GHMDA-2PJ02',\n 'time_start': '2002-07-04T00:00:00.000Z',\n 'title': 'GHRSST Level 2P Global Sea Surface Skin '\n 'Temperature from the Moderate Resolution '\n 'Imaging Spectroradiometer (MODIS) on the NASA '\n 'Aqua satellite (GDS2)',\n 'updated': '2019-12-02T22:59:24.849Z',\n 'version_id': '2019.0'}],\n 'id': 'https://cmr.earthdata.nasa.gov:443/search/collections.json?concept_id=C1940473819-POCLOUD',\n 'title': 'ECHO dataset metadata',\n 'updated': '2021-11-19T17:44:29.026Z'}}\n\n\n\n\nWhat do each of these service values mean?\n\nAssociations\n\nCMR is a large web of interconnected metadata “schemas”, including Collections, Granules, Services, Tools, and Variables. In this case, this collection is associated with two unique services, two tools, and several unique variables.\n\nTags\n\nThere are also tags that describe what service options exist at a high-level. In this case, we see that this dataset supports the ability to reformat, subset by space and time, as well as by variable. This is used in web applications like Earthdata Search to surface those customization options more readily.\n\nService Features\n\nIn this case, we see three separate “features” listed here: esi, Harmony, and OPeNDAP.\n\n\nWe will dig into more details on what Harmony offers for this dataset.\nFirst, we need to isolate the services returned for this dataset:\n\nservices = response['feed']['entry'][0]['associations']['services']\nprint(services)\n\n['S1962070864-POCLOUD', 'S2004184019-POCLOUD']\n\n\n\nservice_url = \"https://cmr.earthdata.nasa.gov/search/services\"\n\nInspect the first service returned. Now we’re going to search the services endpoint to view that individual service’s metadata, like we did with our dataset above. This time, we’re explicitly setting the format of the response to umm-json in the Accept Header in order to see detailed metadata about the service.\n\nservice_response = requests.get(service_url, \n params={\n 'concept_id': services[0],\n },\n headers={\n 'Accept': 'application/vnd.nasa.cmr.umm_results+json'\n }\n )\nservice_response = service_response.json()\n\nDetails about the service metadata record include the service options provided by the “backend” processor connected to Harmony, in this case the PODAAC Level 2 Cloud Subsetter:\n\npprint(service_response)\n\n{'hits': 1,\n 'items': [{'meta': {'concept-id': 'S1962070864-POCLOUD',\n 'concept-type': 'service',\n 'deleted': False,\n 'format': 'application/vnd.nasa.cmr.umm+json',\n 'native-id': 'POCLOUD_podaac_l2_cloud_subsetter',\n 'provider-id': 'POCLOUD',\n 'revision-date': '2021-11-02T22:57:03.597Z',\n 'revision-id': 19,\n 'user-id': 'podaaccloud'},\n 'umm': {'AccessConstraints': 'None',\n 'Description': 'Endpoint for subsetting L2 Subsetter via '\n 'Harmony',\n 'LongName': 'PODAAC Level 2 Cloud Subsetter',\n 'MetadataSpecification': {'Name': 'UMM-S',\n 'URL': 'https://cdn.earthdata.nasa.gov/umm/service/v1.4',\n 'Version': '1.4'},\n 'Name': 'PODAAC L2 Cloud Subsetter',\n 'OperationMetadata': [{'OperationName': 'SPATIAL_SUBSETTING'},\n {'OperationName': 'VARIABLE_SUBSETTING'},\n {'OperationName': 'TEMPORAL_SUBSETTING'}],\n 'ServiceKeywords': [{'ServiceCategory': 'EARTH SCIENCE '\n 'SERVICES',\n 'ServiceTerm': 'SUBSETTING/SUPERSETTING',\n 'ServiceTopic': 'DATA MANAGEMENT/DATA '\n 'HANDLING'}],\n 'ServiceOptions': {'Subset': {'SpatialSubset': {'BoundingBox': {'AllowMultipleValues': False}},\n 'TemporalSubset': {'AllowMultipleValues': False},\n 'VariableSubset': {'AllowMultipleValues': True}},\n 'SupportedReformattings': [{'SupportedInputFormat': 'HDF5',\n 'SupportedOutputFormats': ['NETCDF-4']},\n {'SupportedInputFormat': 'NETCDF-4',\n 'SupportedOutputFormats': ['NETCDF-4']}]},\n 'ServiceOrganizations': [{'LongName': 'Physical '\n 'Oceanography '\n 'Distributed Active '\n 'Archive Center, Jet '\n 'Propulsion '\n 'Laboratory, NASA',\n 'Roles': ['ORIGINATOR'],\n 'ShortName': 'NASA/JPL/PODAAC'}],\n 'Type': 'Harmony',\n 'URL': {'Description': 'PROJECT HOME PAGE',\n 'URLValue': 'https://harmony.earthdata.nasa.gov'},\n 'Version': '1.1.0'}}],\n 'took': 9}\n\n\n\n\nDiscover all datasets that support Harmony services\nInstead of searching for services on a known dataset of interest, we may want to discovery all available datasets that are supported for a given service. We can utilize GraphQL, which is a way for us to efficiently gain information across service and collection metadata so that we can print out all supported Harmony datasets. First, we need to specify a query string. Here we are asking to query all collections with service type “Harmony”, and to provide details on the service options attached to those services:\n\nquery = \"\"\"query {\n collections(limit: 2000, serviceType: \"Harmony\") {\n count\n items {\n shortName\n conceptId\n services {\n count\n items {\n name\n supportedReformattings\n supportedInputProjections\n supportedOutputProjections\n serviceOptions\n }\n }\n variables {\n count\n }\n }\n }\n}\"\"\"\n\nThis utilizes a different API endpoint to query CMR metdata using GraphQL. Here we set up another request, passing our query string above:\n\ngraphql_url = 'https://graphql.earthdata.nasa.gov/api'\n\ngraphql_response = requests.get(graphql_url,\n params={\"query\": query},\n headers={\n 'Accept': 'application/json',\n }\n )\n\nA json response is returned that provides all collections with Harmony-supported services. We can then extract just the collectionshortName, conceptID, and the service names supported for each collection:\n\nservices = graphql_response.json()['data']['collections']['items']\n\nfor service in services:\n print(service['shortName'], \",\", service['conceptId'])\n for i in range(len(service['services']['items'])):\n print(\"Services:\", service['services']['items'][i]['name'])\n\nECCO_L4_ATM_STATE_05DEG_DAILY_V4R4 , C1990404801-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_ATM_STATE_LLC0090GRID_DAILY_V4R4 , C1991543823-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_ATM_STATE_05DEG_MONTHLY_V4R4 , C1990404814-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_ATM_STATE_LLC0090GRID_MONTHLY_V4R4 , C1991543805-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_BOLUS_STREAMFUNCTION_LLC0090GRID_DAILY_V4R4 , C1991543818-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_BOLUS_STREAMFUNCTION_LLC0090GRID_MONTHLY_V4R4 , C1991543733-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_BOLUS_05DEG_DAILY_V4R4 , C1990404807-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_BOLUS_LLC0090GRID_DAILY_V4R4 , C1991543824-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_BOLUS_05DEG_MONTHLY_V4R4 , C1990404805-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_BOLUS_LLC0090GRID_MONTHLY_V4R4 , C1991543745-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_GEOMETRY_05DEG_V4R4 , C2013583732-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_GEOMETRY_LLC0090GRID_V4R4 , C2013557893-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_GMAP_TIME_SERIES_SNAPSHOT_V4R4 , C1991543729-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_GMAP_TIME_SERIES_SNAPSHOT_V4R4B , C2133160276-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_GMSL_TIME_SERIES_DAILY_V4R4 , C1991543819-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_GMSL_TIME_SERIES_MONTHLY_V4R4 , C1991543742-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_MIX_COEFFS_05DEG_V4R4 , C2013584708-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_MIX_COEFFS_LLC0090GRID_V4R4 , C2013583906-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_FRESH_FLUX_05DEG_DAILY_V4R4 , C1990404818-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_FRESH_FLUX_LLC0090GRID_DAILY_V4R4 , C1991543820-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_FRESH_FLUX_05DEG_MONTHLY_V4R4 , C1990404792-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_FRESH_FLUX_LLC0090GRID_MONTHLY_V4R4 , C1991543803-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_HEAT_FLUX_05DEG_DAILY_V4R4 , C1990404788-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_HEAT_FLUX_LLC0090GRID_DAILY_V4R4 , C1991543712-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_HEAT_FLUX_05DEG_MONTHLY_V4R4 , C1990404812-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_HEAT_FLUX_LLC0090GRID_MONTHLY_V4R4 , C1991543811-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_STRESS_05DEG_DAILY_V4R4 , C1990404808-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_STRESS_LLC0090GRID_DAILY_V4R4 , C1991543704-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_STRESS_05DEG_MONTHLY_V4R4 , C1990404796-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_STRESS_LLC0090GRID_MONTHLY_V4R4 , C1991543760-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_05DEG_DAILY_V4R4 , C1990404797-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_05DEG_DAILY_V4R4B , C2129192243-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_LLC0090GRID_DAILY_V4R4 , C1991543737-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_LLC0090GRID_DAILY_V4R4B , C2129195053-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_05DEG_MONTHLY_V4R4 , C1990404791-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_05DEG_MONTHLY_V4R4B , C2129193421-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_LLC0090GRID_MONTHLY_V4R4 , C1991543806-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_LLC0090GRID_MONTHLY_V4R4B , C2129197196-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OBP_LLC0090GRID_SNAPSHOT_V4R4 , C1991543804-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_DENS_STRAT_PRESS_05DEG_DAILY_V4R4 , C1990404793-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_DENS_STRAT_PRESS_LLC0090GRID_DAILY_V4R4 , C1991543727-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_DENS_STRAT_PRESS_05DEG_MONTHLY_V4R4 , C1990404798-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_DENS_STRAT_PRESS_LLC0090GRID_MONTHLY_V4R4 , C1991543735-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_MIXED_LAYER_DEPTH_05DEG_DAILY_V4R4 , C1990404810-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_MIXED_LAYER_DEPTH_LLC0090GRID_DAILY_V4R4 , C1991543734-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_MIXED_LAYER_DEPTH_05DEG_MONTHLY_V4R4 , C1990404819-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_MIXED_LAYER_DEPTH_LLC0090GRID_MONTHLY_V4R4 , C1991543741-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_TEMP_SALINITY_05DEG_DAILY_V4R4 , C1990404821-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_TEMP_SALINITY_LLC0090GRID_DAILY_V4R4 , C1991543736-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_TEMP_SALINITY_05DEG_MONTHLY_V4R4 , C1990404795-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_TEMP_SALINITY_LLC0090GRID_MONTHLY_V4R4 , C1991543728-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_TEMP_SALINITY_LLC0090GRID_SNAPSHOT_V4R4 , C1991543757-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_MOMENTUM_TEND_LLC0090GRID_DAILY_V4R4 , C1991543726-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_MOMENTUM_TEND_LLC0090GRID_MONTHLY_V4R4 , C1991543702-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_TEMPERATURE_FLUX_LLC0090GRID_DAILY_V4R4 , C1991543812-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_TEMPERATURE_FLUX_LLC0090GRID_MONTHLY_V4R4 , C1991543740-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_SALINITY_FLUX_LLC0090GRID_DAILY_V4R4 , C1991543814-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_SALINITY_FLUX_LLC0090GRID_MONTHLY_V4R4 , C1991543752-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_VOLUME_FLUX_LLC0090GRID_DAILY_V4R4 , C1991543699-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_3D_VOLUME_FLUX_LLC0090GRID_MONTHLY_V4R4 , C1991543739-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_VEL_05DEG_DAILY_V4R4 , C1990404811-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_VEL_LLC0090GRID_DAILY_V4R4 , C1991543808-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_VEL_05DEG_MONTHLY_V4R4 , C1990404823-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_OCEAN_VEL_LLC0090GRID_MONTHLY_V4R4 , C1991543732-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SBO_CORE_TIME_SERIES_SNAPSHOT_V4R4 , C1991543766-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SBO_CORE_TIME_SERIES_SNAPSHOT_V4R4B , C2133162585-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_05DEG_DAILY_V4R4 , C1990404813-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_05DEG_DAILY_V4R4B , C2129181904-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_LLC0090GRID_DAILY_V4R4 , C1991543744-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_LLC0090GRID_DAILY_V4R4B , C2129186341-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_05DEG_MONTHLY_V4R4 , C1990404799-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_05DEG_MONTHLY_V4R4B , C2129189405-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_LLC0090GRID_MONTHLY_V4R4 , C1991543813-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_LLC0090GRID_MONTHLY_V4R4B , C2129189870-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SSH_LLC0090GRID_SNAPSHOT_V4R4 , C1991543817-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_CONC_THICKNESS_05DEG_DAILY_V4R4 , C1990404815-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_CONC_THICKNESS_LLC0090GRID_DAILY_V4R4 , C1991543763-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_CONC_THICKNESS_05DEG_MONTHLY_V4R4 , C1990404820-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_CONC_THICKNESS_LLC0090GRID_MONTHLY_V4R4 , C1991543764-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_CONC_THICKNESS_LLC0090GRID_SNAPSHOT_V4R4 , C1991543821-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_HORIZ_VOLUME_FLUX_LLC0090GRID_DAILY_V4R4 , C1991543731-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_HORIZ_VOLUME_FLUX_LLC0090GRID_MONTHLY_V4R4 , C1991543724-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_SALT_PLUME_FLUX_LLC0090GRID_DAILY_V4R4 , C1991543807-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_SALT_PLUME_FLUX_LLC0090GRID_MONTHLY_V4R4 , C1991543730-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_VELOCITY_05DEG_DAILY_V4R4 , C1990404817-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_VELOCITY_LLC0090GRID_DAILY_V4R4 , C1991543765-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_VELOCITY_05DEG_MONTHLY_V4R4 , C1990404790-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_VELOCITY_LLC0090GRID_MONTHLY_V4R4 , C1991543700-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nECCO_L4_SEA_ICE_VELOCITY_LLC0090GRID_SNAPSHOT_V4R4 , C1991543768-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PO.DAAC harmony-netcdf-to-zarr\nVIIRS_NPP-NAVO-L2P-v1.0 , C1996881807-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nVIIRS_NPP-NAVO-L2P-v3.0 , C1996881636-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nMSG03-OSPO-L2P-v1.0 , C2036878029-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nVIIRS_NPP-JPL-L2P-v2016.2 , C1996881456-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nMODIS_A-JPL-L2P-v2019.0 , C1940473819-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nMODIS_A-JPL-L2P-v2019.0 , C1693233348-PODAAC\nServices: PODAAC L2 Cloud Subsetter\nMODIS_T-JPL-L2P-v2019.0 , C1940475563-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nMODIS_T-JPL-L2P-v2019.0 , C1693233387-PODAAC\nServices: PODAAC L2 Cloud Subsetter\nTMI-REMSS-L2P-v4 , C2036879048-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nVIIRS_NPP-OSPO-L2P-v2.61 , C1996880725-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nVIIRS_N20-OSPO-L2P-v2.61 , C1996880450-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nAVHRR_SST_METOP_B-OSISAF-L2P-v1.0 , C2036880717-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nJASON-1_L2_OST_GPR_E , C1684065156-PODAAC\nServices: PODAAC L2 Cloud Subsetter\nJASON-1_L2_OST_GPN_E , C1684065153-PODAAC\nServices: PODAAC L2 Cloud Subsetter\nJASON-1_L2_OST_GPS_E , C1684065158-PODAAC\nServices: PODAAC L2 Cloud Subsetter\nCYGNSS_NOAA_L2_SWSP_25KM_V1.1 , C2036882072-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nJASON_CS_S6A_L2_AMR_RAD_NRT , C1968979997-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nJASON_CS_S6A_L2_AMR_RAD_STC , C1968979762-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nJASON_CS_S6A_L2_ALT_LR_STD_OST_NRT_F , C1968979597-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nJASON_CS_S6A_L2_ALT_LR_RED_OST_NRT_F , C1968980576-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nJASON_CS_S6A_L2_ALT_LR_STD_OST_STC_F , C1968980609-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter\nJASON_CS_S6A_L2_ALT_LR_RED_OST_STC_F , C1968979561-POCLOUD\nServices: PO.DAAC Cloud OPeNDAP\nServices: PODAAC L2 Cloud Subsetter" }, { - "objectID": "external/xarray.html#what-you-will-learn-from-this-tutorial", - "href": "external/xarray.html#what-you-will-learn-from-this-tutorial", - "title": "Introduction to xarray", - "section": "What you will learn from this tutorial", - "text": "What you will learn from this tutorial\nIn this tutorial you will learn how to:\n\nload a netcdf file into xarray\ninterrogate the Dataset and understand the difference between DataArray and Dataset\nsubset a Dataset\ncalculate annual and monthly mean fields\ncalculate a time series of zonal means\nplot these results\n\nAs always, we’ll start by importing xarray. We’ll follow convention by giving the module the shortname xr\n\nimport xarray as xr\nxr.set_options(keep_attrs=True)\n\nI’m going to use one of xarray’s tutorial datasets. In this case, air temperature from the NCEP reanalysis. I’ll assign the result of the open_dataset to ds. I may change this to access a dataset directly\n\nds = xr.tutorial.open_dataset(\"air_temperature\")\n\nAs we are in an interactive environment, we can just type ds to see what we have.\n\nds\n\nFirst thing to notice is that ds is an xarray.Dataset object. It has dimensions, lat, lon, and time. It also has coordinate variables with the same names as these dimensions. These coordinate variables are 1-dimensional. This is a NetCDF convention. The Dataset contains one data variable, air. This has dimensions (time, lat, lon).\nClicking on the document icon reveals attributes for each variable. Clicking on the disk icon reveals a representation of the data.\nEach of the data and coordinate variables can be accessed and examined using the variable name as a key.\n\nds.air\n\n\nds['air']\n\nThese are xarray.DataArray objects. This is the basic building block for xarray.\nVariables can also be accessed as attributes of ds.\n\nds.time\n\nA major difference between accessing a variable as an attribute versus using a key is that the attribute is read-only but the key method can be used to update the variable. For example, if I want to convert the units of air from Kelvin to degrees Celsius.\n\nds['air'] = ds.air - 273.15\n\nThis approach can also be used to add new variables\n\nds['air_kelvin'] = ds.air + 273.15\n\nIt is helpful to update attributes such as units, this saves time, confusion and mistakes, especially when you save the dataset.\n\nds['air'].attrs['units'] = 'degC'\n\n\nds" + "objectID": "external/harmony_subsetting.html#discover-variable-names", + "href": "external/harmony_subsetting.html#discover-variable-names", + "title": "Data Subsetting and Transformation Services in the Cloud", + "section": "Discover variable names", + "text": "Discover variable names\nJust like services, a dataset may also be associated to metadata on their individual data variables, which can be used as input if we wish to subset by one or more variables of interest.\n\nvariables = response['feed']['entry'][0]['associations']['variables']\nprint(variables)\n\n['V1997812737-POCLOUD', 'V1997812697-POCLOUD', 'V2112014688-POCLOUD', 'V1997812756-POCLOUD', 'V1997812688-POCLOUD', 'V1997812670-POCLOUD', 'V1997812724-POCLOUD', 'V2112014684-POCLOUD', 'V1997812701-POCLOUD', 'V1997812681-POCLOUD', 'V2112014686-POCLOUD', 'V1997812663-POCLOUD', 'V1997812676-POCLOUD', 'V1997812744-POCLOUD', 'V1997812714-POCLOUD']\n\n\nSeveral variable records are returned. Again, like we did for services, we’ll search the variables endpoint to view an individual variable’s metadata, and we’ll print out the list of variables for our dataset.\n\nvar_url = \"https://cmr.earthdata.nasa.gov/search/variables\"\n\n\nvar_response = requests.get(var_url, \n params={\n 'concept_id': variables[0],\n },\n headers={\n 'Accept': 'application/vnd.nasa.cmr.umm_results+json'\n }\n )\nvar_response = var_response.json()\n\n\npprint(var_response)\n\n{'hits': 1,\n 'items': [{'associations': {'collections': [{'concept-id': 'C1940473819-POCLOUD'}]},\n 'meta': {'concept-id': 'V1997812737-POCLOUD',\n 'concept-type': 'variable',\n 'deleted': False,\n 'format': 'application/vnd.nasa.cmr.umm+json',\n 'native-id': 'MODIS_A-JPL-L2P-v2019.0-sses_standard_deviation_4um',\n 'provider-id': 'POCLOUD',\n 'revision-date': '2021-10-19T02:26:51.560Z',\n 'revision-id': 6,\n 'user-id': 'jmcnelis'},\n 'umm': {'DataType': 'byte',\n 'Definition': 'mid-IR SST standard deviation error; non '\n 'L2P core field; signed byte array: WARNING '\n 'Some applications are unable to properly '\n 'handle signed byte values. If values are '\n 'encountered > 127, please subtract 256 from '\n 'this reported value',\n 'Dimensions': [{'Name': 'time',\n 'Size': 1,\n 'Type': 'TIME_DIMENSION'},\n {'Name': 'nj',\n 'Size': 2030,\n 'Type': 'ALONG_TRACK_DIMENSION'},\n {'Name': 'ni',\n 'Size': 1354,\n 'Type': 'CROSS_TRACK_DIMENSION'}],\n 'FillValues': [{'Type': 'SCIENCE_FILLVALUE',\n 'Value': -128}],\n 'LongName': 'SSES standard deviation error based on '\n 'proximity confidence flags',\n 'Name': 'sses_standard_deviation_4um',\n 'Offset': 10.0,\n 'Scale': 0.07874016,\n 'Sets': [{'Index': 1,\n 'Name': 'sses_standard_deviation_4um',\n 'Size': 1,\n 'Type': 'General'}],\n 'Units': 'kelvin',\n 'ValidRanges': [{'Max': 127, 'Min': -127}],\n 'VariableType': 'ANCILLARY_VARIABLE'}}],\n 'took': 9}\n\n\nNext, print out a simple list of all associated variable names by looping the same variable response we submitted above, this time for each variable:\n\nvar_list = []\nfor i in range(len(variables)):\n var_response = requests.get(var_url, \n params={\n 'concept_id': variables[i],\n },\n headers={\n 'Accept': 'application/vnd.nasa.cmr.umm_results+json'\n }\n )\n var_response = var_response.json()\n var_list.append(var_response['items'][0]['umm']['Name'])\n\n\npprint(var_list)\n\n['sses_standard_deviation_4um',\n 'l2p_flags',\n 'time',\n 'dt_analysis',\n 'sses_standard_deviation',\n 'sst_dtime',\n 'sses_bias_4um',\n 'lat',\n 'sea_surface_temperature_4um',\n 'sses_bias',\n 'lon',\n 'sea_surface_temperature',\n 'quality_level',\n 'wind_speed',\n 'quality_level_4um']" }, { - "objectID": "external/xarray.html#subsetting-and-indexing", - "href": "external/xarray.html#subsetting-and-indexing", - "title": "Introduction to xarray", - "section": "Subsetting and Indexing", - "text": "Subsetting and Indexing\nSubsetting and indexing methods depend on whether you are working with a Dataset or DataArray. A DataArray can be accessed using positional indexing just like a numpy array. To access the temperature field for the first time step, you do the following.\n\nds['air'][0,:,:]\n\nNote this returns a DataArray with coordinates but not attributes.\nHowever, the real power is being able to access variables using coordinate variables. I can get the same subset using the following. (It’s also more explicit about what is being selected and robust in case I modify the DataArray and expect the same output.)\n\nds['air'].sel(time='2013-01-01').time\n\n\nds.air.sel(time='2013-01-01')\n\nI can also do slices. I’ll extract temperatures for the state of Colorado. The bounding box for the state is [-109 E, -102 E, 37 N, 41 N].\nIn the code below, pay attention to both the order of the coordinates and the range of values. The first value of the lat coordinate variable is 41 N, the second value is 37 N. Unfortunately, xarray expects slices of coordinates to be in the same order as the coordinates. Note lon is 0 to 360 not -180 to 180, and I let python calculate it for me within the slice.\n\nds.air.sel(lat=slice(41.,37.), lon=slice(360-109,360-102))\n\nWhat if we want temperature for a point, for example Denver, CO (39.72510678889283 N, -104.98785545855408 E). xarray can handle this! If we just want data from the nearest grid point, we can use sel and specify the method as “nearest”.\n\ndenver_lat, denver_lon = 39.72510678889283, -104.98785545855408\n\n\nds.air.sel(lat=denver_lat, lon=360+denver_lon, method='nearest')\n\nIf we want to interpolate, we can use interp(). In this case I use linear or bilinear interpolation.\ninterp() can also be used to resample data to a new grid and even reproject data\n\nds.air.interp(lat=denver_lat, lon=360+denver_lon, method='linear')\n\nsel() and interp() can also be used on Dataset objects.\n\nds.sel(lat=slice(41.,37.), lon=slice(360-109,360-102))\n\n\nds.interp(lat=denver_lat, lon=360+denver_lon, method='linear')" + "objectID": "external/harmony_subsetting.html#using-harmony-py-to-subset-data", + "href": "external/harmony_subsetting.html#using-harmony-py-to-subset-data", + "title": "Data Subsetting and Transformation Services in the Cloud", + "section": "Using Harmony-Py to subset data", + "text": "Using Harmony-Py to subset data\nHarmony-Py provides a pip installable Python alternative to directly using Harmony’s RESTful API to make it easier to request data and service options, especially when interacting within a Python Jupyter Notebook environment.\nThe next steps are adopted from the introduction tutorial notebook provided in the Harmony-Py library:\n\nCreate Harmony Client object\nFirst, we need to create a Harmony Client, which is what we will interact with to submit and inspect a data request to Harmony, as well as to retrieve results.\nWhen creating the Client, we need to provide Earthdata Login credentials, which are required to access data from NASA EOSDIS. This basic line below assumes that we have a .netrc available.\n\nharmony_client = Client()\n\n\n\nHurricane Ida snapshot\nUsing NASA Worldview, we can first explore SST during a tropical storm event; in this case, we can overlay L2 and L4 SST variables against true color imagery to observe Hurricane Ida in August 2021. Although this is a small sample set, this use case could be expanded to explore how SST responds during the Atlantic hurricane over the next several months. The same data that we are requesting below using Harmony-py can also be requested using NASA Earthdata Search\n\n\n\nHurrican Ida snapshot - Worldview\n\n\n\n\nCreate Harmony Request\nThe following are common request parameters:\n\ncollection: Required parameter. This is the NASA EOSDIS collection, or data product. There are two options for inputting a collection of interest:\n\nProvide a concept ID (e.g. C1940473819-POCLOUD)\nData product short name (e.g. MODIS_A-JPL-L2P-v2019.0).\n\nspatial: Bounding box spatial constraints on the data. The Harmony Bbox class accepts spatial coordinates as decimal degrees in w, s, e, n order, where longitude = -180, 180 and latitude = -90, 90.\ntemporal: Date/time constraints on the data. The example below demonstrates temporal start and end ranges using the python datetime library.\n\nAs we identified above, only subsetting options are available for this dataset. If other service options such as reformatting are available for a given dataset, these can also be specified using Harmony-py: See the documentation for details on how to construct these parameters.\n\nrequest = Request(\n collection=Collection(id=short_name),\n spatial=BBox(-97.77667,21.20806,-83.05197,30.16605),\n temporal={\n 'start': dt.datetime(2021, 8, 20),\n 'stop': dt.datetime(2021, 8, 21),\n },\n)\n\n\n\nCheck Request validity\nBefore submitting a Harmony Request, we can test your request to see if it’s valid and how to fix it if not. In particular, request.is_valid will check to ensure that the spatial BBox bounds and temporal ranges are entered correctly.\n\nrequest.is_valid()\n\nTrue\n\n\n\n\nSubmit request\nNow that the request is created, we can now submit it to Harmony using the Harmony Client object. A job id is returned, which is a unique identifier that represents the submitted request.\n\njob_id = harmony_client.submit(request)\njob_id\n\n'e36f6916-a7d9-4d82-a5fa-924d48d464ef'\n\n\n\n\nCheck request status\nWe can check on the progress of a processing job with status(). This method blocks while communicating with the server but returns quickly.\n\nharmony_client.status(job_id)\n\n{'status': 'running',\n 'message': 'There were 2 collections that matched the provided short name MODIS_A-JPL-L2P-v2019.0. See https://cmr.earthdata.nasa.gov/concepts/C1940473819-POCLOUD for details on the selected collection. The version ID for the selected collection is 2019.0. To use a different collection submit a new request specifying the desired CMR concept ID instead of the collection short name.',\n 'progress': 0,\n 'created_at': datetime.datetime(2021, 11, 19, 17, 44, 40, 768000, tzinfo=tzlocal()),\n 'updated_at': datetime.datetime(2021, 11, 19, 17, 44, 40, 768000, tzinfo=tzlocal()),\n 'request': 'https://harmony.earthdata.nasa.gov/MODIS_A-JPL-L2P-v2019.0/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(21.20806%3A30.16605)&subset=lon(-97.77667%3A-83.05197)&subset=time(%222021-08-20T00%3A00%3A00%22%3A%222021-08-21T00%3A00%3A00%22)',\n 'num_input_granules': 6}\n\n\nDepending on the size of the request, it may be helpful to wait until the request has completed processing before the remainder of the code is executed. The wait_for_processing() method will block subsequent lines of code while optionally showing a progress bar.\n\nharmony_client.wait_for_processing(job_id, show_progress=True)\n\n [ Processing: 100% ] |###################################################| [|]\n\n\n\n\nView Harmony job response and output URLs\nOnce the data request has finished processing, we can view details on the job that was submitted to Harmony, including the API call to Harmony, and informational messages on the request if available.\nresult_json() calls wait_for_processing() and returns the complete job in JSON format once processing is complete.\n\ndata = harmony_client.result_json(job_id)\npprint(data)\n\n{'createdAt': '2021-11-19T17:44:40.768Z',\n 'jobID': 'e36f6916-a7d9-4d82-a5fa-924d48d464ef',\n 'links': [{'href': 'https://harmony.earthdata.nasa.gov/stac/e36f6916-a7d9-4d82-a5fa-924d48d464ef/',\n 'rel': 'stac-catalog-json',\n 'title': 'STAC catalog',\n 'type': 'application/json'},\n {'bbox': [-83.612, 1.103, -58.391, 22.005],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820062501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T06:29:58.000Z',\n 'start': '2021-08-20T06:25:01.000Z'},\n 'title': '20210820062501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-97.8, 21.2, -83.1, 30.2],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820080001-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T08:04:58.000Z',\n 'start': '2021-08-20T08:00:01.000Z'},\n 'title': '20210820080001-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-128.335, 29.006, -95.854, 49.87],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820093501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T09:39:49.000Z',\n 'start': '2021-08-20T09:35:01.000Z'},\n 'title': '20210820093501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-96.8, 21.2, -83.1, 27.7],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820185501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T18:59:58.000Z',\n 'start': '2021-08-20T18:55:01.000Z'},\n 'title': '20210820185501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-97.8, 25.3, -83.1, 30.2],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820190001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T19:04:58.000Z',\n 'start': '2021-08-20T19:00:01.000Z'},\n 'title': '20210820190001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-97.8, 21.2, -97.3, 24.6],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820203501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T20:39:58.000Z',\n 'start': '2021-08-20T20:35:01.000Z'},\n 'title': '20210820203501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'href': 'https://harmony.earthdata.nasa.gov/jobs/e36f6916-a7d9-4d82-a5fa-924d48d464ef?linktype=https&page=1&limit=2000',\n 'rel': 'self',\n 'title': 'The current page',\n 'type': 'application/json'}],\n 'message': 'There were 2 collections that matched the provided short name '\n 'MODIS_A-JPL-L2P-v2019.0. See '\n 'https://cmr.earthdata.nasa.gov/concepts/C1940473819-POCLOUD for '\n 'details on the selected collection. The version ID for the '\n 'selected collection is 2019.0. To use a different collection '\n 'submit a new request specifying the desired CMR concept ID '\n 'instead of the collection short name.',\n 'numInputGranules': 6,\n 'progress': 100,\n 'request': 'https://harmony.earthdata.nasa.gov/MODIS_A-JPL-L2P-v2019.0/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(21.20806%3A30.16605)&subset=lon(-97.77667%3A-83.05197)&subset=time(%222021-08-20T00%3A00%3A00%22%3A%222021-08-21T00%3A00%3A00%22)',\n 'status': 'successful',\n 'updatedAt': '2021-11-19T17:45:17.148Z',\n 'username': 'amy.steiker'}\n\n\n\n\nDirect cloud access\nNote that the remainder of this tutorial will only succeed when running this notebook within the AWS us-west-2 region.\nHarmony data outputs can be accessed within the cloud using the s3 URLs and AWS credentials provided in the Harmony job response.\n\nRetrieve list of output URLs.\nThe result_urls() method calls wait_for_processing() and returns a list of the processed data URLs once processing is complete. You may optionally show the progress bar as shown below.\n\nresults = harmony_client.result_urls(job_id, link_type=LinkType.s3)\nurls = list(results)\npprint(urls)\n\n['s3://harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820062501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820080001-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0_subsetted.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820093501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820185501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820190001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820203501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0_subsetted.nc4']\n\n\nWe can see that the first file returned does not include the _subsetted suffix, which indicates that a blank file was returned, as no data values were located within our subsetted region. We’ll select the second URL in the list to bring into xarray below.\n\nurl = urls[1]\nurl\n\n's3://harmony-prod-staging/public/podaac/l2-subsetter/cef7a7c7-c01c-4186-ac73-3e03f2940259/20210820080001-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0_subsetted.nc4'\n\n\n\n\nAWS credential retrieval\nUsing aws_credentials you can retrieve the credentials needed to access the Harmony s3 staging bucket and its contents.\n\ncreds = harmony_client.aws_credentials()\n\n\n\n\nOpen staged files with s3fs and xarray\nWe use the AWS s3fs package to create a file system that can then be read by xarray:\n\ns3_fs = s3fs.S3FileSystem(\n key=creds['aws_access_key_id'],\n secret=creds['aws_secret_access_key'],\n token=creds['aws_session_token'],\n client_kwargs={'region_name':'us-west-2'},\n)\n\nNow that we have our s3 file system set, including our declared credentials, we’ll use that to open the url, and read in the file through xarray. This extra step is needed because xarray cannot open the S3 location directly. Instead, the S3 file object is passed to xarray, in order to then open the dataset.\n\nf = s3_fs.open(url, mode='rb')\nds = xr.open_dataset(f)\nds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (nj: 1232, ni: 1132, time: 1)\nCoordinates:\n lat (nj, ni, time) float32 ...\n lon (nj, ni, time) float32 ...\n * time (time) datetime64[ns] 2021-08-20T08:00:01\nDimensions without coordinates: nj, ni\nData variables:\n sea_surface_temperature (time, nj, ni) float32 ...\n sst_dtime (time, nj, ni) timedelta64[ns] ...\n quality_level (time, nj, ni) float32 ...\n sses_bias (time, nj, ni) float32 ...\n sses_standard_deviation (time, nj, ni) float32 ...\n l2p_flags (time, nj, ni) int16 ...\n sea_surface_temperature_4um (time, nj, ni) float32 ...\n quality_level_4um (time, nj, ni) float32 ...\n sses_bias_4um (time, nj, ni) float32 ...\n sses_standard_deviation_4um (time, nj, ni) float32 ...\n wind_speed (time, nj, ni) float32 ...\n dt_analysis (time, nj, ni) float32 ...\nAttributes: (12/50)\n Conventions: CF-1.7, ACDD-1.3\n title: MODIS Aqua L2P SST\n summary: Sea surface temperature retrievals produced a...\n references: GHRSST Data Processing Specification v2r5\n institution: NASA/JPL/OBPG/RSMAS\n history: MODIS L2P created at JPL PO.DAAC\\n2021-11-19 ...\n ... ...\n processing_level: L2P\n cdm_data_type: swath\n startDirection: Descending\n endDirection: Descending\n day_night_flag: Night\n history_json: [{\"date_time\": \"2021-11-19T17:44:56.897126+00...xarray.DatasetDimensions:nj: 1232ni: 1132time: 1Coordinates: (3)lat(nj, ni, time)float32...long_name :latitudestandard_name :latitudeunits :degrees_northvalid_min :[-90.]valid_max :[90.]comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[1394624 values with dtype=float32]lon(nj, ni, time)float32...long_name :longitudestandard_name :longitudeunits :degrees_eastvalid_min :[-180.]valid_max :[180.]comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[1394624 values with dtype=float32]time(time)datetime64[ns]2021-08-20T08:00:01long_name :reference time of sst filestandard_name :timecomment :time of first sensor observationcoverage_content_type :coordinatearray(['2021-08-20T08:00:01.000000000'], dtype='datetime64[ns]')Data variables: (12)sea_surface_temperature(time, nj, ni)float32...long_name :sea surface temperaturestandard_name :sea_surface_skin_temperatureunits :kelvinvalid_min :[-1000]valid_max :[10000]comment :sea surface temperature from thermal IR (11 um) channelssource :NASA and University of Miamicoverage_content_type :physicalMeasurement[1394624 values with dtype=float32]sst_dtime(time, nj, ni)timedelta64[ns]...long_name :time difference from reference timevalid_min :[-32767]valid_max :[32767]comment :time plus sst_dtime gives seconds after 00:00:00 UTC January 1, 1981coverage_content_type :referenceInformation[1394624 values with dtype=timedelta64[ns]]quality_level(time, nj, ni)float32...long_name :quality level of SST pixelvalid_min :[0]valid_max :[5]comment :thermal IR SST proximity confidence value; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valueflag_values :[0 1 2 3 4 5]flag_meanings :no_data bad_data worst_quality low_quality acceptable_quality best_qualitycoverage_content_type :qualityInformation[1394624 values with dtype=float32]sses_bias(time, nj, ni)float32...long_name :SSES bias error based on proximity confidence flagsunits :kelvinvalid_min :[-127]valid_max :[127]comment :thermal IR SST bias error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[1394624 values with dtype=float32]sses_standard_deviation(time, nj, ni)float32...long_name :SSES standard deviation error based on proximity confidence flagsunits :kelvinvalid_min :[-127]valid_max :[127]comment :thermal IR SST standard deviation error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[1394624 values with dtype=float32]l2p_flags(time, nj, ni)int16...long_name :L2P flagsvalid_min :[0]valid_max :[16]comment :These flags can be used to further filter data variablesflag_meanings :microwave land ice lake riverflag_masks :[ 1 2 4 8 16]coverage_content_type :qualityInformation[1394624 values with dtype=int16]sea_surface_temperature_4um(time, nj, ni)float32...long_name :sea surface temperatureunits :kelvinvalid_min :[-1000]valid_max :[10000]comment :sea surface temperature from mid-IR (4 um) channels; non L2P core fieldcoverage_content_type :physicalMeasurement[1394624 values with dtype=float32]quality_level_4um(time, nj, ni)float32...long_name :quality level of SST pixelvalid_min :[0]valid_max :[5]comment :mid-IR SST proximity confidence value; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valueflag_values :[0 1 2 3 4 5]flag_meanings :no_data bad_data worst_quality low_quality acceptable_quality best_qualitycoverage_content_type :qualityInformation[1394624 values with dtype=float32]sses_bias_4um(time, nj, ni)float32...long_name :SSES bias error based on proximity confidence flagsunits :kelvinvalid_min :[-127]valid_max :[127]comment :mid-IR SST bias error; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[1394624 values with dtype=float32]sses_standard_deviation_4um(time, nj, ni)float32...long_name :SSES standard deviation error based on proximity confidence flagsunits :kelvinvalid_min :[-127]valid_max :[127]comment :mid-IR SST standard deviation error; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[1394624 values with dtype=float32]wind_speed(time, nj, ni)float32...long_name :10m wind speedstandard_name :wind_speedunits :m s-1valid_min :[-127]valid_max :[127]comment :Wind at 10 meters above the sea surfacesource :TBD. Placeholder. Currently emptygrid_mapping :TBDtime_offset :[2.]height :10 mcoverage_content_type :auxiliaryInformation[1394624 values with dtype=float32]dt_analysis(time, nj, ni)float32...long_name :deviation from SST reference climatologyunits :kelvinvalid_min :[-127]valid_max :[127]comment :TBDsource :TBD. Placeholder. Currently emptycoverage_content_type :auxiliaryInformation[1394624 values with dtype=float32]Attributes: (50)Conventions :CF-1.7, ACDD-1.3title :MODIS Aqua L2P SSTsummary :Sea surface temperature retrievals produced at the NASA OBPG for the MODIS Aqua sensor. These have been reformatted to GHRSST GDS specifications by the JPL PO.DAACreferences :GHRSST Data Processing Specification v2r5institution :NASA/JPL/OBPG/RSMAShistory :MODIS L2P created at JPL PO.DAAC\n2021-11-19 17:44:56.897087 l2ss-py v1.1.0 (bbox=[[-97.77667, -83.05197], [21.20806, 30.16605]] cut=True)comment :L2P Core without DT analysis or other ancillary fields; Night, Start Node:Descending, End Node:Descending; WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value; Quicklooklicense :GHRSST and PO.DAAC protocol allow data use as free and open.id :MODIS_A-JPL-L2P-v2019.0naming_authority :org.ghrsstproduct_version :2019.0uuid :f6e1f61d-c4a4-4c17-8354-0c15e12d688bgds_version_id :2.0netcdf_version_id :4.1date_created :20210820T110619Zfile_quality_level :[3]spatial_resolution :1kmstart_time :20210820T080001Ztime_coverage_start :20210820T080001Zstop_time :20210820T080458Ztime_coverage_end :20210820T080458Znorthernmost_latitude :[35.9455]southernmost_latitude :[14.8953]easternmost_longitude :[-78.2345]westernmost_longitude :[-105.765]source :MODIS sea surface temperature observations for the OBPGplatform :Aquasensor :MODISmetadata_link :http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MODIS_A-JPL-L2P-v2019.0keywords :Oceans > Ocean Temperature > Sea Surface Temperaturekeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordsstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventiongeospatial_lat_units :degrees_northgeospatial_lat_resolution :[0.01]geospatial_lon_units :degrees_eastgeospatial_lon_resolution :[0.01]acknowledgment :The MODIS L2P sea surface temperature data are sponsored by NASAcreator_name :Ed Armstrong, JPL PO.DAACcreator_email :edward.m.armstrong@jpl.nasa.govcreator_url :http://podaac.jpl.nasa.govproject :Group for High Resolution Sea Surface Temperaturepublisher_name :The GHRSST Project Officepublisher_url :http://www.ghrsst.orgpublisher_email :ghrsst-po@nceo.ac.ukprocessing_level :L2Pcdm_data_type :swathstartDirection :DescendingendDirection :Descendingday_night_flag :Nighthistory_json :[{\"date_time\": \"2021-11-19T17:44:56.897126+00:00\", \"derived_from\": \"https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20210820080001-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\", \"program\": \"l2ss-py\", \"version\": \"1.1.0\", \"parameters\": \"bbox=[[-97.77667, -83.05197], [21.20806, 30.16605]] cut=True\", \"program_ref\": \"https://cmr.earthdata.nasa.gov:443/search/concepts/S1962070864-POCLOUD\", \"$schema\": \"https://harmony.earthdata.nasa.gov/schemas/history/0.1.0/history-v0.1.0.json\"}]\n\n\n\n\nPlot the data\nUse the xarray built in plotting function to create a simple plot along the x and y dimensions of the dataset:\n\nds.sea_surface_temperature.plot() ;" }, { - "objectID": "external/xarray.html#analysis", - "href": "external/xarray.html#analysis", - "title": "Introduction to xarray", - "section": "Analysis", - "text": "Analysis\nAs a simple example, let’s try to calculate a mean field for the whole time range.\n\nds.mean(dim='time')\n\nWe can also calculate a zonal mean (averaging over longitude)\n\nds.mean(dim='lon')\n\nOther aggregation methods include min(), max(), std(), along with others.\n\nds.std(dim='time')\n\nThe data we have are in 6h timesteps. This can be resampled to daily or monthly. If you are familiar with pandas, xarray uses the same methods.\n\nds.resample(time='M').mean()\n\n\nds_mon = ds.resample(time='M').mean()\nds_mon\n\nThis is a really short time series but as an example, let’s calculate a monthly climatology (at least for 2 months). For this we can use groupby()\n\nds_clim = ds_mon.groupby(ds_mon.time.dt.month).mean()" + "objectID": "external/harmony_subsetting.html#resources", + "href": "external/harmony_subsetting.html#resources", + "title": "Data Subsetting and Transformation Services in the Cloud", + "section": "Resources", + "text": "Resources\n\nIn-depth exploration of the MODIS_A-JPL-L2P-v2019.0 data set, co-locating in-situ and remote sensing data: https://github.com/podaac/tutorials/blob/master/notebooks/SWOT-EA-2021/Colocate_satellite_insitu_ocean.ipynb\nHarmony-Py library introduction tutorial: https://github.com/nasa/harmony-py/blob/main/examples/intro_tutorial.ipynb" }, { - "objectID": "external/xarray.html#plot-results", - "href": "external/xarray.html#plot-results", - "title": "Introduction to xarray", - "section": "Plot results", - "text": "Plot results\nFinally, let’s plot the results! This will plot the lat/lon axes of the original ds DataArray.\n\nds_clim.air.sel(month=10).plot()" + "objectID": "external/zarr-eosdis-store.html", + "href": "external/zarr-eosdis-store.html", + "title": "Zarr Example", + "section": "", + "text": "imported on: 2023-05-30\n\nThis notebook is from NASA’s Zarr EOSDIS store notebook\n\n\nThe original source for this document is https://github.com/nasa/zarr-eosdis-store\n\n\nzarr-eosdis-store example\nInstall dependencies\n\nimport sys\n\n# zarr and zarr-eosdis-store, the main libraries being demoed\n!{sys.executable} -m pip install zarr zarr-eosdis-store\n\n# Notebook-specific libraries\n!{sys.executable} -m pip install matplotlib\n\nImportant: To run this, you must first create an Earthdata Login account (https://urs.earthdata.nasa.gov) and place your credentials in ~/.netrc e.g.:\n machine urs.earthdata.nasa.gov login YOUR_USER password YOUR_PASSWORD\nNever share or commit your password / .netrc file!\nBasic usage. After these lines, we work with ds as though it were a normal Zarr dataset\n\nimport zarr\nfrom eosdis_store import EosdisStore\n\nurl = 'https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20210715090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc'\n\nds = zarr.open(EosdisStore(url))\n\nView the file’s variable structure\n\nprint(ds.tree())\n\n/\n ├── analysed_sst (1, 17999, 36000) int16\n ├── analysis_error (1, 17999, 36000) int16\n ├── dt_1km_data (1, 17999, 36000) int16\n ├── lat (17999,) float32\n ├── lon (36000,) float32\n ├── mask (1, 17999, 36000) int16\n ├── sea_ice_fraction (1, 17999, 36000) int16\n ├── sst_anomaly (1, 17999, 36000) int16\n └── time (1,) int32\n\n\nFetch the latitude and longitude arrays and determine start and end indices for our area of interest. In this case, we’re looking at the Great Lakes, which have a nice, recognizeable shape. Latitudes 41 to 49, longitudes -93 to 76.\n\nlats = ds['lat'][:]\nlons = ds['lon'][:]\nlat_range = slice(lats.searchsorted(41), lats.searchsorted(49))\nlon_range = slice(lons.searchsorted(-93), lons.searchsorted(-76))\n\nGet the analysed sea surface temperature variable over our area of interest and apply scale factor and offset from the file metadata. In a future release, scale factor and add offset will be automatically applied.\n\nvar = ds['analysed_sst']\nanalysed_sst = var[0, lat_range, lon_range] * var.attrs['scale_factor'] + var.attrs['add_offset']\n\nDraw a pretty picture\n\nfrom matplotlib import pyplot as plt\n\nplt.rcParams[\"figure.figsize\"] = [16, 8]\nplt.imshow(analysed_sst[::-1, :])\nNone\n\n\n\n\nIn a dozen lines of code and a few seconds, we have managed to fetch and visualize the 3.2 megabyte we needed from a 732 megabyte file using the original archive URL and no processing services" }, { - "objectID": "external/on-prem_cloud.html", - "href": "external/on-prem_cloud.html", - "title": "Pairing Cloud and non-Cloud Data", + "objectID": "external/appeears_csv_cloud_access.html", + "href": "external/appeears_csv_cloud_access.html", + "title": "How to work with AppEEARS Point Sample CSV outputs", "section": "", - "text": "imported on: 2023-05-30\nThis notebook was originally developed for the 2021 Cloud Hackathon, and has been updated with most current approaches." + "text": "This tutorial demonstrates how to access AppEEARS point sample Comma-Separated Values (CSV) outputs direcly from the cloud. NASA’s Application for Extracting and Exploring Analysis Ready Samples (AρρEEARS) has been migrated to NASA’s Earthdata Cloud space located in AWS us-west 2. This enables the user working in the cloud instance deployed in AWS us-west 2 to access outputs direcly in the cloud using S3 link returned in the location header of the response. In this tutorial, we will walk through the process of submitting a point sample and accessing a CSV outputs from AppEEARS.\nRequirements\n- Earthdata Login Authentication is required to access AppEEARS API and AppEEARS outpurs direcrly from an Amazon AWS bucket. See Requirements section in README.md.\nLearning Objectives\n- Learn how to access AppEEARS point sample CSV outputs direcly from the cloud.\nTutorial Outline\n1. Setting up\n2. Submit a point request\n3. Extract the S3 links to data in S3\n4. Create a boto3 Refreshable Session\n5. Direct S3 access of CSV output\n6. Quality Filtering\n7. Explore the LST time series" }, { - "objectID": "external/on-prem_cloud.html#accessing-and-harmonizing-data-located-within-and-outside-of-the-nasa-earthdata-cloud", - "href": "external/on-prem_cloud.html#accessing-and-harmonizing-data-located-within-and-outside-of-the-nasa-earthdata-cloud", - "title": "Pairing Cloud and non-Cloud Data", - "section": "Accessing and harmonizing data located within and outside of the NASA Earthdata Cloud", - "text": "Accessing and harmonizing data located within and outside of the NASA Earthdata Cloud" + "objectID": "external/appeears_csv_cloud_access.html#setting-up", + "href": "external/appeears_csv_cloud_access.html#setting-up", + "title": "How to work with AppEEARS Point Sample CSV outputs", + "section": "1. Setting up", + "text": "1. Setting up\nImport required packages.\n\nimport requests\nimport numpy\nimport datetime\nimport os\nimport requests\nfrom netrc import netrc\nfrom pathlib import Path\nfrom datetime import datetime, timezone\nfrom botocore.client import Config\nimport pandas\nimport time\nimport warnings\nimport sys\nsys.path.append('../Python/modules/')\nimport aws_session\nwarnings.filterwarnings('ignore')\n\nTo successfully run this tutorial, it is required to create a .netrc file in your home directory. The function _validate_netrc defined in aws_session checks if a properly formatted netrc file exists in your home directory. If the netrc file does not exist, it will prompt you for your Earthdata Login username and password and will create a netrc file. Please see the Prerequisites section in README.md.\n\n# validate if netrc file is present else create one via the user / password prompt for the urs.earthdata.nasa.gov\naws_session._validate_netrc()\n\nINFO : netrc file is setup for urs.earthdata.nasa.gov already ..." }, { - "objectID": "external/on-prem_cloud.html#timing", - "href": "external/on-prem_cloud.html#timing", - "title": "Pairing Cloud and non-Cloud Data", - "section": "Timing", - "text": "Timing\n\nExercise: 45 min" + "objectID": "external/appeears_csv_cloud_access.html#submit-a-point-request-in-appeears", + "href": "external/appeears_csv_cloud_access.html#submit-a-point-request-in-appeears", + "title": "How to work with AppEEARS Point Sample CSV outputs", + "section": "2. Submit a point request in AppEEARS", + "text": "2. Submit a point request in AppEEARS\nIn this step, we are going to submit a point request. You can also submit this same request to the AppEEARS Graphic User Interface (GUI) by uploading the JSON file provided in the repository (AppEEARS-Data-Resources/Data/point-request.json). If you have completed the request, save your task_id to a variable, skip this step, and move to the next step of tutorial.\nAssign the AρρEEARS API endpoint to a variable.\n\nappeears_API_endpoint = 'https://appeears.earthdatacloud.nasa.gov/api/'\n\nA Bearer Token is needed to submit requests to the AppEEARS API. To generated a token, a POST request containing Earthdata Login credentials stored in .netrc file is submitted to the login service from the AppEEARS API.\n\nurs = 'urs.earthdata.nasa.gov'\n\n\ntoken_response = requests.post('{}login'.format(appeears_API_endpoint), auth = (netrc().authenticators(urs)[0],netrc().authenticators(urs)[2])).json() # Insert API URL, call login service, provide credentials & return json \ntoken_response\n\n{'token_type': 'Bearer',\n 'token': 'BL2uWT50_qVKOTtcDr2yigaNm-83GN0A1W8lYmiihzSVl2BIoHXnESxNmAMjNWDIGyIwsZusAtcoaxYSpmpl4A',\n 'expiration': '2023-03-15T15:13:36Z'}\n\n\nWe’ll save the token to as an Authorization object we can pass in the header of any request made to the AppEEARS API.\n\ntoken = token_response['token'] # Save login token to a variable\nhead = {'Authorization': 'Bearer {}'.format(token)} # Create a header to store token information, needed to submit a request\n\nThe payload for an AppEEARS API request is a JSON object. Here, we compile the request JSON object with the required parameters. The coordinates (i.e., lat/lon pair) for where the LP DAAC is located are used as our point location. The data layers LST_Night_1km and LST_Day_1km from MOD11A1 version 6.1 are selected below.\n\ntask_name = \"EROS\"\ntask_type = 'point' # Type of task, area or point\nstartDate = '01-01-2015' # Start of the date range for which to extract data: MM-DD-YYYY\nendDate = '03-01-2023' # End of the date range for which to extract data: MM-DD-YYYY\nlat = [43.736038]\nlon = [-96.62139]\nprodLayer = [\n {\"layer\": \"LST_Day_1km\", \"product\": \"MOD11A1.061\"}, \n {\"layer\": \"LST_Night_1km\", \"product\": \"MOD11A1.061\"}\n]\n\n\ntask = {\n 'task_type': task_type,\n 'task_name': task_name,\n 'params': {\n 'dates': [\n {\n 'startDate': startDate,\n 'endDate': endDate\n }],\n 'layers': prodLayer,\n 'coordinates': [\n {\n 'latitude': lat[0],\n 'longitude': lon[0],\n 'id': \"EROS\"\n \n }]\n \n }\n}\n\nNext, submit the AppEEARS request using post function from requests library.\n\ntask_response = requests.post('{}task'.format(appeears_API_endpoint), json=task, headers=head).json() # Post json to the API task service, return response as json\ntask_response\n\n{'task_id': '9b2f9a77-da1a-41ac-ba47-baeb577ad99f', 'status': 'pending'}\n\n\nSave the task_id and wait until your request is processed and complete.\n\ntask_id = task_response['task_id']\ntask_id\n\n'9b2f9a77-da1a-41ac-ba47-baeb577ad99f'\n\n\n\n# Ping API until request is complete, then continue to Section 3\nwhile requests.get('{}task/{}'.format(appeears_API_endpoint, task_id), headers=head).json()['status'] != 'done':\n print(requests.get('{}task/{}'.format(appeears_API_endpoint, task_id), headers=head).json()['status'])\n time.sleep(60)\nprint(requests.get('{}task/{}'.format(appeears_API_endpoint, task_id), headers=head).json()['status'])\n\nqueued\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\nprocessing\ndone" }, { - "objectID": "external/on-prem_cloud.html#summary", - "href": "external/on-prem_cloud.html#summary", - "title": "Pairing Cloud and non-Cloud Data", - "section": "Summary", - "text": "Summary\nThis tutorial will combine several workflow steps and components from the previous days, demonstrating the process of using the geolocation of data available outside of the Earthdata Cloud to then access coincident variables of cloud-accessible data. This may be a common use case as NASA Earthdata continues to migrate to the cloud, producing a “hybrid” data archive across Amazon Web Services (AWS) and original on-premise data storage systems. Additionally, you may also want to combine field measurements with remote sensing data available on the Earthdata Cloud.\nThis specific example explores the pairing of the ICESat-2 ATL07 Sea Ice Height data product, currently (as of November 2021) available publicly via direct download at the NSIDC DAAC, along with Sea Surface Temperature (SST) from the GHRSST MODIS L2 dataset (MODIS_A-JPL-L2P-v2019.0) available from PO.DAAC on the Earthdata Cloud.\nThe use case we’re looking at today centers over an area north of Greenland for a single day in June, where a melt pond was observed using the NASA OpenAltimetry application. Melt ponds are an important feature of Arctic sea ice dynamics, leading to an decrease in sea ice albedo and other changes in heat balance. Many NASA Earthdata datasets produce variables including sea ice albedo, sea surface temperature, air temperature, and sea ice height, which can be used to better understand these dynamics.\n\nObjectives\n\nPractice skills searching for data in CMR, determining granule coverage across two datasets over an area of interest.\nDownload data from an on-premise storage system to our cloud environment.\nRead in 1-dimensional trajectory data (ICESat-2 ATL07) into xarray and perform attribute conversions.\nSelect and read in sea surface temperature (SST) data (MODIS_A-JPL-L2P-v2019.0) from the Earthdata Cloud into xarray.\nExtract, resample, and plot coincident SST data based on ICESat-2 geolocation." + "objectID": "external/appeears_csv_cloud_access.html#extract-the-direct-s3-links", + "href": "external/appeears_csv_cloud_access.html#extract-the-direct-s3-links", + "title": "How to work with AppEEARS Point Sample CSV outputs", + "section": "3. Extract the Direct S3 links", + "text": "3. Extract the Direct S3 links\nNow that we have our outputs ready, we can get the bundle information for the files included in the outputs. If you submitted your request using AppEEARS GUI, assign your sample’s task_id to the variable task_id below.\n\n#task_id = '<ADD task_id / request_id from AppEEARS GUI>'\n\nrequests.get is used toget the bundle information. Below, bundle information is printed. The bundle information includes s3_url in addition to the other information such as file_name, file_id, and file_type.\nEach output file can be downloaded using the file_id and AppEEARS API endpoint. AppEEARS outputs are also stored in an AWS bucket that can be accessed using S3_url.\n\nbundle = requests.get('{}bundle/{}'.format(appeears_API_endpoint,task_id), headers=head).json() # Call API and return bundle contents for the task_id as json\nbundle\n\n{'files': [{'sha256': '2d217eb9d967d849212bf6702a91e3fd615a7188cba72bf58e0b4d9849417865',\n 'file_id': '959e4c02-6ff1-4370-ba0d-67f5613eb3fa',\n 'file_name': 'EROS-MOD11A1-061-results.csv',\n 'file_size': 1168238,\n 'file_type': 'csv',\n 's3_url': 's3://appeears-output/9b2f9a77-da1a-41ac-ba47-baeb577ad99f/EROS-MOD11A1-061-results.csv'},\n {'sha256': '70dc8abc16c8368f6926f735f5f9057c27338a41f77165e648c600185acdff39',\n 'file_id': 'd198e3ac-93ce-4587-92df-f296c3d6b75b',\n 'file_name': 'EROS-granule-list.txt',\n 'file_size': 479520,\n 'file_type': 'txt',\n 's3_url': 's3://appeears-output/9b2f9a77-da1a-41ac-ba47-baeb577ad99f/EROS-granule-list.txt'},\n {'sha256': 'dd9359b792376d6939fac09b8be74ad5db8ca74e009aa386325f4d5ff0b1a004',\n 'file_id': '7c0280db-6764-4672-a58f-e8cc29229c32',\n 'file_name': 'EROS-request.json',\n 'file_size': 786,\n 'file_type': 'json',\n 's3_url': 's3://appeears-output/9b2f9a77-da1a-41ac-ba47-baeb577ad99f/EROS-request.json'},\n {'sha256': '3795b274a5aaceade145441ca09fe9a73925c64ef32d235923e66112007e142e',\n 'file_id': 'd1f49518-357a-45b7-b6ee-4b83411f1389',\n 'file_name': 'EROS-MOD11A1-061-metadata.xml',\n 'file_size': 16775,\n 'file_type': 'xml',\n 's3_url': 's3://appeears-output/9b2f9a77-da1a-41ac-ba47-baeb577ad99f/EROS-MOD11A1-061-metadata.xml'},\n {'sha256': '551b41b89b0a1130a00a6e35f4d6812f284253fe37f994cefeec18b92ce8967d',\n 'file_id': 'b926d235-70d0-4b63-aa5a-5abe32a032d4',\n 'file_name': 'README.md',\n 'file_size': 18943,\n 'file_type': 'txt',\n 's3_url': 's3://appeears-output/9b2f9a77-da1a-41ac-ba47-baeb577ad99f/README.md'}],\n 'created': '2023-03-13T16:05:31.183063',\n 'task_id': '9b2f9a77-da1a-41ac-ba47-baeb577ad99f',\n 'updated': '2023-03-13T16:41:31.639875',\n 'bundle_type': 'point'}\n\n\nBelow, the S3 Links to CSV output is filted.\n\nfile = [f for f in bundle['files'] if f['file_type'] == 'csv']\ncsv_urls = file[0]['s3_url']" }, { - "objectID": "external/on-prem_cloud.html#import-packages", - "href": "external/on-prem_cloud.html#import-packages", - "title": "Pairing Cloud and non-Cloud Data", - "section": "Import packages", - "text": "Import packages\n\nimport os\nfrom pathlib import Path\nfrom pprint import pprint\n\n# Access EDS\nimport requests\n\n# Access AWS S3\nimport s3fs\n\n# Read and work with datasets\nimport xarray as xr\nimport numpy as np\nimport h5py\n\n# For plotting\nimport matplotlib.pyplot as plt\nimport cartopy.crs as ccrs\nfrom shapely.geometry import box\n\n# For resampling\nimport pyresample" + "objectID": "external/appeears_csv_cloud_access.html#create-a-boto3-refreshable-session", + "href": "external/appeears_csv_cloud_access.html#create-a-boto3-refreshable-session", + "title": "How to work with AppEEARS Point Sample CSV outputs", + "section": "4. Create a boto3 Refreshable Session", + "text": "4. Create a boto3 Refreshable Session\nAppEEARS outputs are freely accessible from a cloud instance in us-west-2 region. In order to access our output files, a Boto3 session is needed. The Boto session will stores the required configurations for an easy integration between Python and AWS services. Below, get_boto3_refreshable_session stored in aws_session will access your Earthdata login credentidals store in .netrc file and generate S3 credential by making a call to AppEEARS S3 credential endpoint, and create a boto3 session. This session will be auto-renewed as needed to prevent timeouts errors related to S3 credentials.\n\nregion_name = 'us-west-2'\ns3_creds_endpoint = f\"{appeears_API_endpoint}/s3credentials\"\n# Boto3 Session required by the pandas package \nboto3_session = aws_session.get_boto3_refreshable_session(s3_creds_endpoint, region_name)\n\nINFO : Getting new temporary credentials from AppEEARS API https://appeears.earthdatacloud.nasa.gov/api//s3credentials...\n\n\nNext, an object for s3 client is created using boto3_session.\n\nboto3_client = boto3_session.client('s3', config=Config(max_pool_connections=20))" }, { - "objectID": "external/on-prem_cloud.html#specify-data-time-range-and-area-of-interest", - "href": "external/on-prem_cloud.html#specify-data-time-range-and-area-of-interest", - "title": "Pairing Cloud and non-Cloud Data", - "section": "Specify data, time range, and area of interest", - "text": "Specify data, time range, and area of interest\nWe are going to focus on getting data for an area north of Greenland for a single day in June.\nThese bounding_box and temporal variables will be used for data search, subset, and access below:\n\n# Bounding Box spatial parameter in decimal degree 'W,S,E,N' format.\nbounding_box = '-62.8,81.7,-56.4,83'\n\n# Each date in yyyy-MM-ddTHH:mm:ssZ format; date range in start,end format\ntemporal = '2019-06-22T00:00:00Z,2019-06-22T23:59:59Z'\n\nSince we’ve already demonstrated how to locate a dataset’s collection_id and use the cloud_hosted parameter to determine whether a dataset resides in the Earthdata Cloud, we are going to skip forward and declare these variables:\n\nmodis_concept_id = 'C1940473819-POCLOUD'\nicesat2_concept_id = 'C2003771980-NSIDC_ECS'" + "objectID": "external/appeears_csv_cloud_access.html#csv-file-in-region-direct-s3-access", + "href": "external/appeears_csv_cloud_access.html#csv-file-in-region-direct-s3-access", + "title": "How to work with AppEEARS Point Sample CSV outputs", + "section": "5. CSV File In-Region Direct S3 Access", + "text": "5. CSV File In-Region Direct S3 Access\nTo read the CSV file, an object is created for the bucket name along with the file name of csv file. AppEEARS stores the output csv file in a folder. Bucket and key can be extracted using S3 URL as below.\n\nbucket_name = csv_urls.split('/')[2]\nkey_name = csv_urls.split(bucket_name)[1].strip(\"/\")\nobj = boto3_client.get_object(Bucket = bucket_name, Key = key_name) \n\nNext, read_csv from pandas is used to read the body of created object.\n\noutput_csv = pandas.read_csv(obj['Body'])\noutput_csv\n\n\n\n\n\n\n\n\nID\nLatitude\nLongitude\nDate\nMODIS_Tile\nMOD11A1_061_Line_Y_1km\nMOD11A1_061_Sample_X_1km\nMOD11A1_061_LST_Day_1km\nMOD11A1_061_LST_Night_1km\nMOD11A1_061_QC_Day\n...\nMOD11A1_061_QC_Day_LST_Error_Flag_Description\nMOD11A1_061_QC_Night_bitmask\nMOD11A1_061_QC_Night_MODLAND\nMOD11A1_061_QC_Night_MODLAND_Description\nMOD11A1_061_QC_Night_Data_Quality_flag\nMOD11A1_061_QC_Night_Data_Quality_flag_Description\nMOD11A1_061_QC_Night_Emis_Error_flag\nMOD11A1_061_QC_Night_Emis_Error_flag_Description\nMOD11A1_061_QC_Night_LST_Error_Flag\nMOD11A1_061_QC_Night_LST_Error_Flag_Description\n\n\n\n\n0\nEROS\n43.736038\n-96.62139\n2015-01-01\nh11v04\n751.0\n22.0\n0.000000\n0.000000\n2.0\n...\nAverage LST error <= 1K\n0b00000010\n0b10\nLST not produced due to cloud effects\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n1\nEROS\n43.736038\n-96.62139\n2015-01-02\nh11v04\n751.0\n22.0\n268.940002\n0.000000\n65.0\n...\nAverage LST error <= 2K\n0b00000010\n0b10\nLST not produced due to cloud effects\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n2\nEROS\n43.736038\n-96.62139\n2015-01-03\nh11v04\n751.0\n22.0\n0.000000\n0.000000\n2.0\n...\nAverage LST error <= 1K\n0b00000010\n0b10\nLST not produced due to cloud effects\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n3\nEROS\n43.736038\n-96.62139\n2015-01-04\nh11v04\n751.0\n22.0\n256.259979\n0.000000\n65.0\n...\nAverage LST error <= 2K\n0b00000010\n0b10\nLST not produced due to cloud effects\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n4\nEROS\n43.736038\n-96.62139\n2015-01-05\nh11v04\n751.0\n22.0\n0.000000\n0.000000\n2.0\n...\nAverage LST error <= 1K\n0b00000010\n0b10\nLST not produced due to cloud effects\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n...\n\n\n2955\nEROS\n43.736038\n-96.62139\n2023-02-25\nh11v04\n751.0\n22.0\n264.540008\n0.000000\n0.0\n...\nAverage LST error <= 1K\n0b00000010\n0b10\nLST not produced due to cloud effects\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n2956\nEROS\n43.736038\n-96.62139\n2023-02-26\nh11v04\n751.0\n22.0\n270.299988\n257.759979\n0.0\n...\nAverage LST error <= 1K\n0b00000000\n0b00\nLST produced, good quality, not necessary to e...\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n2957\nEROS\n43.736038\n-96.62139\n2023-02-27\nh11v04\n751.0\n22.0\n0.000000\n0.000000\n2.0\n...\nAverage LST error <= 1K\n0b00000010\n0b10\nLST not produced due to cloud effects\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n2958\nEROS\n43.736038\n-96.62139\n2023-02-28\nh11v04\n751.0\n22.0\n0.000000\n267.239990\n2.0\n...\nAverage LST error <= 1K\n0b00000000\n0b00\nLST produced, good quality, not necessary to e...\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b00\nAverage LST error <= 1K\n\n\n2959\nEROS\n43.736038\n-96.62139\n2023-03-01\nh11v04\n751.0\n22.0\n0.000000\n270.239990\n2.0\n...\nAverage LST error <= 1K\n0b01000001\n0b01\nLST produced, other quality, recommend examina...\n0b00\nGood data quality\n0b00\nAverage emissivity error <= 0.01\n0b01\nAverage LST error <= 2K\n\n\n\n\n2960 rows × 29 columns" }, { - "objectID": "external/on-prem_cloud.html#search-and-download-icesat-2-atl07-files", - "href": "external/on-prem_cloud.html#search-and-download-icesat-2-atl07-files", - "title": "Pairing Cloud and non-Cloud Data", - "section": "Search and download ICESat-2 ATL07 files", - "text": "Search and download ICESat-2 ATL07 files\nPerform a granule search over our time and area of interest. How many granules are returned?\n\ngranule_url = 'https://cmr.earthdata.nasa.gov/search/granules'\n\n\nresponse = requests.get(granule_url,\n params={\n 'concept_id': icesat2_concept_id,\n 'temporal': temporal,\n 'bounding_box': bounding_box,\n 'page_size': 200,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\nprint(response.headers['CMR-Hits'])\n\n2\n\n\nPrint the file names, size, and links:\n\ngranules = response.json()['feed']['entry']\nfor granule in granules:\n print(f'{granule[\"producer_granule_id\"]} {granule[\"granule_size\"]} {granule[\"links\"][0][\"href\"]}')\n\nATL07-01_20190622055317_12980301_004_01.h5 237.0905504227 https://n5eil01u.ecs.nsidc.org/DP9/ATLAS/ATL07.004/2019.06.22/ATL07-01_20190622055317_12980301_004_01.h5\nATL07-01_20190622200154_13070301_004_01.h5 230.9151573181 https://n5eil01u.ecs.nsidc.org/DP9/ATLAS/ATL07.004/2019.06.22/ATL07-01_20190622200154_13070301_004_01.h5\n\n\n\nDownload ATL07 files\nAlthough several services are supported for ICESat-2 data, we are demonstrating direct access through the “on-prem” file system at NSIDC for simplicity.\nSome of these services include: - icepyx - From the icepyx documentation: “icepyx is both a software library and a community composed of ICESat-2 data users, developers, and the scientific community. We are working together to develop a shared library of resources - including existing resources, new code, tutorials, and use-cases/examples - that simplify the process of querying, obtaining, analyzing, and manipulating ICESat-2 datasets to enable scientific discovery.” - NSIDC DAAC Data Access and Service API - The API provided by the NSIDC DAAC allows you to access data programmatically using specific temporal and spatial filters. The same subsetting, reformatting, and reprojection services available on select data sets through NASA Earthdata Search can also be applied using this API. - IceFlow - The IceFlow python library simplifies accessing and combining data from several of NASA’s cryospheric altimetry missions, including ICESat/GLAS, Operation IceBridge, and ICESat-2. In particular, IceFlow harmonizes the various file formats and georeferencing parameters across several of the missions’ data sets, allowing you to analyze data across the multi-decadal time series.\nWe’ve found 2 granules. We’ll download the first one and write it to a file with the same name as the producer_granule_id.\nWe need the url for the granule as well. This is href links we printed out above.\n\nicesat_id = granules[0]['producer_granule_id']\nicesat_url = granules[0]['links'][0]['href']\n\nTo retrieve the granule data, we use the requests.get() method, which will utilize the .netrc file on the backend to authenticate the request against Earthdata Login.\n\nr = requests.get(icesat_url)\n\nThe response returned by requests has the same structure as all the other responses: a header and contents. The header information has information about the response, including the size of the data we downloaded in bytes.\n\nfor k, v in r.headers.items():\n print(f'{k}: {v}')\n\nDate: Thu, 18 Nov 2021 04:02:03 GMT\nServer: Apache\nVary: User-Agent\nContent-Disposition: attachment\nContent-Length: 248607461\nKeep-Alive: timeout=15, max=100\nConnection: Keep-Alive\n\n\nThe contents needs to be saved to a file. To keep the directory clean, we will create a downloads directory to store the file. We can use a shell command to do this or use the makedirs method from the os package.\n\nos.makedirs(\"downloads\", exist_ok=True)\n\nYou should see a downloads directory in the file browser.\nTo write the data to a file, we use open to open a file. We need to specify that the file is open for writing by using the write-mode w. We also need to specify that we want to write bytes by setting the binary-mode b. This is important because the response contents are bytes. The default mode for open is text-mode. So make sure you use b.\nWe’ll use the with statement context-manager to open the file, write the contents of the response, and then close the file. Once the data in r.content is written sucessfully to the file, or if there is an error, the file is closed by the context-manager.\nWe also need to prepend the downloads path to the filename. We do this using Path from the pathlib package in the standard library.\n\noutfile = Path('downloads', icesat_id)\n\n\nif not outfile.exists():\n with open(outfile, 'wb') as f:\n f.write(r.content)\n\nATL07-01_20190622055317_12980301_004_01.h5 is an HDF5 file. xarray can open this but you need to tell it which group to read the data from. In this case we read the sea ice segment height data for ground-track 1 left-beam. You can explore the variable hierarchy in Earthdata Search, by selecting the Customize option under Download Data.\nThis code block performs the following operations: - Extracts the height_segment_height variable from the heights group, along with the dimension variables contained in the higher level sea_ice_segments group, - Convert attributes from bytestrings to strings, - Drops the HDF attribute DIMENSION_LIST, - Sets _FillValue to NaN\n\nvariable_names = [\n '/gt1l/sea_ice_segments/latitude',\n '/gt1l/sea_ice_segments/longitude',\n '/gt1l/sea_ice_segments/delta_time',\n '/gt1l/sea_ice_segments/heights/height_segment_height'\n ]\nwith h5py.File(outfile, 'r') as h5:\n data_vars = {}\n for varname in variable_names:\n var = h5[varname]\n name = varname.split('/')[-1]\n # Convert attributes\n attrs = {}\n for k, v in var.attrs.items():\n if k != 'DIMENSION_LIST':\n if isinstance(v, bytes):\n attrs[k] = v.decode('utf-8')\n else:\n attrs[k] = v\n data = var[:]\n if '_FillValue' in attrs:\n data = np.where(data < attrs['_FillValue'], data, np.nan)\n data_vars[name] = (['segment'], data, attrs)\n is2_ds = xr.Dataset(data_vars)\n \nis2_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (segment: 235584)\nDimensions without coordinates: segment\nData variables:\n latitude (segment) float64 82.38 82.38 82.38 ... 72.61 72.61\n longitude (segment) float64 -55.11 -55.11 ... 145.1 145.1\n delta_time (segment) float64 4.642e+07 4.642e+07 ... 4.642e+07\n height_segment_height (segment) float32 nan nan nan ... -0.4335 -0.4463xarray.DatasetDimensions:segment: 235584Coordinates: (0)Data variables: (4)latitude(segment)float6482.38 82.38 82.38 ... 72.61 72.61contentType :referenceInformationcoordinates :delta_time longitudedescription :Latitude, WGS84, North=+, Lat of segment centerlong_name :Latitudesource :ATBD, section 4.4standard_name :latitudeunits :degrees_northvalid_max :90.0valid_min :-90.0array([82.38431982, 82.38431982, 82.38431982, ..., 72.60984638,\n 72.60977493, 72.60970985])longitude(segment)float64-55.11 -55.11 ... 145.1 145.1contentType :referenceInformationcoordinates :delta_time latitudedescription :Longitude, WGS84, East=+,Lon of segment centerlong_name :Longitudesource :ATBD, section 4.4standard_name :longitudeunits :degrees_eastvalid_max :180.0valid_min :-180.0array([-55.10896068, -55.10896068, -55.10896068, ..., 145.05396164,\n 145.05392851, 145.05389832])delta_time(segment)float644.642e+07 4.642e+07 ... 4.642e+07CLASS :DIMENSION_SCALENAME :gt1l/sea_ice_segments/delta_timeREFERENCE_LIST :[(<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)]contentType :physicalMeasurementcoordinates :latitude longitudedescription :Number of GPS seconds since the ATLAS SDP epoch. The ATLAS Standard Data Products (SDP) epoch offset is defined within /ancillary_data/atlas_sdp_gps_epoch as the number of GPS seconds between the GPS epoch (1980-01-06T00:00:00.000000Z UTC) and the ATLAS SDP epoch. By adding the offset contained within atlas_sdp_gps_epoch to delta time parameters, the time in gps_seconds relative to the GPS epoch can be computed.long_name :Elapsed GPS secondssource :telemetrystandard_name :timeunits :seconds since 2018-01-01array([46419293.64266939, 46419293.64266939, 46419293.64266939, ...,\n 46419681.87646231, 46419681.87759533, 46419681.87862704])height_segment_height(segment)float32nan nan nan ... -0.4335 -0.4463_FillValue :3.4028235e+38contentType :referenceInformationcoordinates :../delta_time ../latitude ../longitudedescription :Mean height from along-track segment fit detremined by the sea ice algorithm. The sea ice height is relative to the tide-free MSS.long_name :height of segment surfacesource :ATBD, section 4.2.2.4units :metersarray([ nan, nan, nan, ..., -0.46550068,\n -0.43347716, -0.4462675 ], dtype=float32)Attributes: (0)\n\n\n\nis2_ds.height_segment_height.plot() ;" + "objectID": "external/appeears_csv_cloud_access.html#quality-filtering", + "href": "external/appeears_csv_cloud_access.html#quality-filtering", + "title": "How to work with AppEEARS Point Sample CSV outputs", + "section": "6. Quality Filtering", + "text": "6. Quality Filtering\nTo get the good quality data, the quality unique values are assigned to a variable. Good quality data have a decoded quality flag as LST produced, good quality, not necessary to examine more detailed QA. The LST values equal to zero, the layer fill_value, are also filtered.\n\nquality_val = numpy.array(output_csv['MOD11A1_061_QC_Night_MODLAND_Description'])\nquality_unique = numpy.unique(quality_val)\nquality_unique\n\narray(['LST not produced due to cloud effects',\n 'LST produced, good quality, not necessary to examine more detailed QA',\n 'LST produced, other quality, recommend examination of more detailed QA'],\n dtype=object)\n\n\n\ngood_qual_flag = 'LST produced, good quality, not necessary to examine more detailed QA'\ngoodQual = output_csv[output_csv['MOD11A1_061_QC_Night_MODLAND_Description'] == good_qual_flag]\ngoodQual = goodQual[goodQual['MOD11A1_061_LST_Day_1km'] != 0]" }, { - "objectID": "external/on-prem_cloud.html#determine-the-ghrsst-modis-l2-granules-returned-from-our-time-and-area-of-interest", - "href": "external/on-prem_cloud.html#determine-the-ghrsst-modis-l2-granules-returned-from-our-time-and-area-of-interest", - "title": "Pairing Cloud and non-Cloud Data", - "section": "Determine the GHRSST MODIS L2 granules returned from our time and area of interest", - "text": "Determine the GHRSST MODIS L2 granules returned from our time and area of interest\n\nresponse = requests.get(granule_url, \n params={\n 'concept_id': modis_concept_id,\n 'temporal': temporal,\n 'bounding_box': bounding_box,\n 'page_size': 200,\n },\n headers={\n 'Accept': 'application/json'\n }\n )\nprint(response.headers['CMR-Hits'])\n\n14\n\n\n\ngranules = response.json()['feed']['entry']\nfor granule in granules:\n print(f'{granule[\"title\"]} {granule[\"granule_size\"]} {granule[\"links\"][0][\"href\"]}')\n\n20190622000501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 20.71552562713623 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622000501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622014501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 9.34600830078125E-5 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622014501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622032501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 21.307741165161133 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622032501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622050001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 9.34600830078125E-5 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622050001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622050501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 20.065649032592773 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622050501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622064001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 9.34600830078125E-5 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622064001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622064501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0 18.602201461791992 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622064501-JPL-L2P_GHRSST-SSTskin-MODIS_A-N-v02.0-fv01.0.nc\n20190622064501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 18.665077209472656 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622064501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622082001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 19.782299995422363 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622082001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622100001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 19.13440227508545 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622100001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622113501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 20.3239164352417 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622113501-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622114001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 9.34600830078125E-5 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622114001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622163001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 19.257243156433105 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622163001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc\n20190622181001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0 19.93498420715332 s3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622181001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc" + "objectID": "external/appeears_csv_cloud_access.html#explore-the-lst-time-series", + "href": "external/appeears_csv_cloud_access.html#explore-the-lst-time-series", + "title": "How to work with AppEEARS Point Sample CSV outputs", + "section": "6. Explore the LST Time Series", + "text": "6. Explore the LST Time Series\nThe matplotlib can be used to create visualizations for a Pandas DataFrames. pyplot is used below to visualize non-Nan day and night LST observations flagged with the good quality.\n\nimport matplotlib.pyplot as plt\n\n\nfig, ax1 = plt.subplots(figsize=(15, 8))\n\nax1.plot(goodQual['Date'], goodQual['MOD11A1_061_LST_Day_1km'] - 273.15, color=\"#6495ED\", lw=2)\nax1.plot(goodQual['Date'], goodQual['MOD11A1_061_LST_Night_1km'] - 273.15, color=\"#CD3333\", lw=2)\n\nax1.set_xlabel(\"Date\")\nax1.set_ylabel(\"Land Surface Temperature (Celsius °)\", fontsize=14)\n\nax1.set_xticks (goodQual['Date'][::14]) # print pone label for every 14 obs \n\nfig.suptitle(\"Day and Night LST\", fontsize=20)\nfig.autofmt_xdate()\n\nINFO : Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.\nINFO : Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.\nINFO : Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.\nINFO : Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.\nINFO : Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting." }, { - "objectID": "external/on-prem_cloud.html#load-data-into-xarray-via-s3-direct-access", - "href": "external/on-prem_cloud.html#load-data-into-xarray-via-s3-direct-access", - "title": "Pairing Cloud and non-Cloud Data", - "section": "Load data into xarray via S3 direct access", - "text": "Load data into xarray via S3 direct access\nOur CMR granule search returned 14 files for our time and area of interest. However, not all granules will be suitable for analysis.\nI’ve identified the image with granule id G1956158784-POCLOUD as a good candidate, this is the 9th granule. In this image, our area of interest is close to nadir. This means that the instantaneous field of view over the area of interest cover a smaller area than at the edge of the image.\nWe are looking for the link for direct download access via s3. This is a url but with a prefix s3://. This happens to be the first href link in the metadata.\nFor a single granule we can cut and paste the s3 link. If we have several granules, the s3 links can be extracted with some simple code.\n\ngranule = granules[9]\n\nfor link in granule['links']:\n if link['href'].startswith('s3://'):\n s3_link = link['href']\n \ns3_link\n\n's3://podaac-ops-cumulus-protected/MODIS_A-JPL-L2P-v2019.0/20190622100001-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc'\n\n\n\nGet S3 credentials\nAs with the previous S3 download tutorials we need credentials to access data from s3: access keys and tokens.\n\ns3_credentials = requests.get('https://archive.podaac.earthdata.nasa.gov/s3credentials').json()\n\nEssentially, what we are doing in this step is to “mount” the s3 bucket as a file system. This allows us to treat the S3 bucket in a similar way to a local file system.\n\ns3_fs = s3fs.S3FileSystem(\n key=s3_credentials[\"accessKeyId\"],\n secret=s3_credentials[\"secretAccessKey\"],\n token=s3_credentials[\"sessionToken\"],\n)\n\n\n\nOpen a s3 file\nNow we have the S3FileSystem set up, we can access the granule. xarray cannot open a S3File directly, so we use the open method for the S3FileSystem to open the granule using the endpoint url we extracted from the metadata. We also have to set the mode='rb'. This opens the granule in read-only mode and in byte-mode. Byte-mode is important. By default, open opens a file as text - in this case it would just be a string of characters - and xarray doesn’t know what to do with that.\nWe then pass the S3File object f to xarray.open_dataset. For this dataset, we also have to set decode_cf=False. This switch tells xarray not to use information contained in variable attributes to generate human readable coordinate variables. Normally, this should work for netcdf files but for this particular cloud-hosted dataset, variable attribute data is not in the form expected by xarray. We’ll fix this.\n\nf = s3_fs.open(s3_link, mode='rb')\nmodis_ds = xr.open_dataset(f, decode_cf=False)\n\nIf you click on the Show/Hide Attributes icon (the first document-like icon to the right of coordinate variable metadata) you can see that attributes are one-element arrays containing bytestrings.\n\nmodis_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (nj: 2030, ni: 1354, time: 1)\nCoordinates:\n * time (time) int32 1214042401\nDimensions without coordinates: nj, ni\nData variables:\n lat (nj, ni) float32 ...\n lon (nj, ni) float32 ...\n sea_surface_temperature (time, nj, ni) int16 ...\n sst_dtime (time, nj, ni) int16 ...\n quality_level (time, nj, ni) int8 ...\n sses_bias (time, nj, ni) int8 ...\n sses_standard_deviation (time, nj, ni) int8 ...\n l2p_flags (time, nj, ni) int16 ...\n chlorophyll_a (time, nj, ni) float32 ...\n K_490 (time, nj, ni) int16 ...\n wind_speed (time, nj, ni) int8 ...\n dt_analysis (time, nj, ni) int8 ...\nAttributes: (12/49)\n Conventions: [b'CF-1.7, ACDD-1.3']\n title: [b'MODIS Aqua L2P SST']\n summary: [b'Sea surface temperature retrievals produce...\n references: [b'GHRSST Data Processing Specification v2r5']\n institution: [b'NASA/JPL/OBPG/RSMAS']\n history: [b'MODIS L2P created at JPL PO.DAAC']\n ... ...\n publisher_email: [b'ghrsst-po@nceo.ac.uk']\n processing_level: [b'L2P']\n cdm_data_type: [b'swath']\n startDirection: [b'Ascending']\n endDirection: [b'Descending']\n day_night_flag: [b'Day']xarray.DatasetDimensions:nj: 2030ni: 1354time: 1Coordinates: (1)time(time)int321214042401long_name :[b'reference time of sst file']standard_name :[b'time']units :[b'seconds since 1981-01-01 00:00:00']comment :[b'time of first sensor observation']coverage_content_type :[b'coordinate']array([1214042401], dtype=int32)Data variables: (12)lat(nj, ni)float32...long_name :[b'latitude']standard_name :[b'latitude']units :[b'degrees_north']_FillValue :[-999.]valid_min :[-90.]valid_max :[90.]comment :[b'geographical coordinates, WGS84 projection']coverage_content_type :[b'coordinate'][2748620 values with dtype=float32]lon(nj, ni)float32...long_name :[b'longitude']standard_name :[b'longitude']units :[b'degrees_east']_FillValue :[-999.]valid_min :[-180.]valid_max :[180.]comment :[b'geographical coordinates, WGS84 projection']coverage_content_type :[b'coordinate'][2748620 values with dtype=float32]sea_surface_temperature(time, nj, ni)int16...long_name :[b'sea surface temperature']standard_name :[b'sea_surface_skin_temperature']units :[b'kelvin']_FillValue :[-32767]valid_min :[-1000]valid_max :[10000]comment :[b'sea surface temperature from thermal IR (11 um) channels']scale_factor :[0.005]add_offset :[273.15]source :[b'NASA and University of Miami']coordinates :[b'lon lat']coverage_content_type :[b'physicalMeasurement'][2748620 values with dtype=int16]sst_dtime(time, nj, ni)int16...long_name :[b'time difference from reference time']units :[b'seconds']_FillValue :[-32768]valid_min :[-32767]valid_max :[32767]comment :[b'time plus sst_dtime gives seconds after 00:00:00 UTC January 1, 1981']coordinates :[b'lon lat']coverage_content_type :[b'referenceInformation'][2748620 values with dtype=int16]quality_level(time, nj, ni)int8...long_name :[b'quality level of SST pixel']_FillValue :[-128]valid_min :[0]valid_max :[5]comment :[b'thermal IR SST proximity confidence value; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value']coordinates :[b'lon lat']flag_values :[0 1 2 3 4 5]flag_meanings :[b'no_data bad_data worst_quality low_quality acceptable_quality best_quality']coverage_content_type :[b'qualityInformation'][2748620 values with dtype=int8]sses_bias(time, nj, ni)int8...long_name :[b'SSES bias error based on proximity confidence flags']units :[b'kelvin']_FillValue :[-128]valid_min :[-127]valid_max :[127]comment :[b'thermal IR SST bias error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value']scale_factor :[0.15748031]add_offset :[0.]coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int8]sses_standard_deviation(time, nj, ni)int8...long_name :[b'SSES standard deviation error based on proximity confidence flags']units :[b'kelvin']_FillValue :[-128]valid_min :[-127]valid_max :[127]comment :[b'thermal IR SST standard deviation error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value']scale_factor :[0.07874016]add_offset :[10.]coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int8]l2p_flags(time, nj, ni)int16...long_name :[b'L2P flags']valid_min :[0]valid_max :[16]comment :[b'These flags can be used to further filter data variables']coordinates :[b'lon lat']flag_meanings :[b'microwave land ice lake river']flag_masks :[ 1 2 4 8 16]coverage_content_type :[b'qualityInformation'][2748620 values with dtype=int16]chlorophyll_a(time, nj, ni)float32...long_name :[b'Chlorophyll Concentration, OC3 Algorithm']units :[b'mg m^-3']_FillValue :[-32767.]valid_min :[0.001]valid_max :[100.]comment :[b'non L2P core field']coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=float32]K_490(time, nj, ni)int16...long_name :[b'Diffuse attenuation coefficient at 490 nm (OBPG)']units :[b'm^-1']_FillValue :[-32767]valid_min :[50]valid_max :[30000]comment :[b'non L2P core field']scale_factor :[0.0002]add_offset :[0.]coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int16]wind_speed(time, nj, ni)int8...long_name :[b'10m wind speed']standard_name :[b'wind_speed']units :[b'm s-1']_FillValue :[-128]valid_min :[-127]valid_max :[127]comment :[b'Wind at 10 meters above the sea surface']scale_factor :[0.2]add_offset :[25.]source :[b'TBD. Placeholder. Currently empty']coordinates :[b'lon lat']grid_mapping :[b'TBD']time_offset :[2.]height :[b'10 m']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int8]dt_analysis(time, nj, ni)int8...long_name :[b'deviation from SST reference climatology']units :[b'kelvin']_FillValue :[-128]valid_min :[-127]valid_max :[127]comment :[b'TBD']scale_factor :[0.1]add_offset :[0.]source :[b'TBD. Placeholder. Currently empty']coordinates :[b'lon lat']coverage_content_type :[b'auxiliaryInformation'][2748620 values with dtype=int8]Attributes: (49)Conventions :[b'CF-1.7, ACDD-1.3']title :[b'MODIS Aqua L2P SST']summary :[b'Sea surface temperature retrievals produced at the NASA OBPG for the MODIS Aqua sensor. These have been reformatted to GHRSST GDS specifications by the JPL PO.DAAC']references :[b'GHRSST Data Processing Specification v2r5']institution :[b'NASA/JPL/OBPG/RSMAS']history :[b'MODIS L2P created at JPL PO.DAAC']comment :[b'L2P Core without DT analysis or other ancillary fields; Day, Start Node:Ascending, End Node:Descending; WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value; Refined']license :[b'GHRSST and PO.DAAC protocol allow data use as free and open.']id :[b'MODIS_A-JPL-L2P-v2019.0']naming_authority :[b'org.ghrsst']product_version :[b'2019.0']uuid :[b'f6e1f61d-c4a4-4c17-8354-0c15e12d688b']gds_version_id :[b'2.0']netcdf_version_id :[b'4.1']date_created :[b'20200221T085224Z']file_quality_level :[3]spatial_resolution :[b'1km']start_time :[b'20190622T100001Z']time_coverage_start :[b'20190622T100001Z']stop_time :[b'20190622T100459Z']time_coverage_end :[b'20190622T100459Z']northernmost_latitude :[89.9862]southernmost_latitude :[66.2723]easternmost_longitude :[-45.9467]westernmost_longitude :[152.489]source :[b'MODIS sea surface temperature observations for the OBPG']platform :[b'Aqua']sensor :[b'MODIS']metadata_link :[b'http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MODIS_A-JPL-L2P-v2019.0']keywords :[b'Oceans > Ocean Temperature > Sea Surface Temperature']keywords_vocabulary :[b'NASA Global Change Master Directory (GCMD) Science Keywords']standard_name_vocabulary :[b'NetCDF Climate and Forecast (CF) Metadata Convention']geospatial_lat_units :[b'degrees_north']geospatial_lat_resolution :[0.01]geospatial_lon_units :[b'degrees_east']geospatial_lon_resolution :[0.01]acknowledgment :[b'The MODIS L2P sea surface temperature data are sponsored by NASA']creator_name :[b'Ed Armstrong, JPL PO.DAAC']creator_email :[b'edward.m.armstrong@jpl.nasa.gov']creator_url :[b'http://podaac.jpl.nasa.gov']project :[b'Group for High Resolution Sea Surface Temperature']publisher_name :[b'The GHRSST Project Office']publisher_url :[b'http://www.ghrsst.org']publisher_email :[b'ghrsst-po@nceo.ac.uk']processing_level :[b'L2P']cdm_data_type :[b'swath']startDirection :[b'Ascending']endDirection :[b'Descending']day_night_flag :[b'Day']\n\n\nTo fix this, we need to extract array elements as scalars, and convert those scalars from bytestrings to strings. We use the decode method to do this. The bytestrings are encoded as utf-8, which is a unicode character format. This is the default encoding for decode but we’ve included it as an argument to be explicit.\nNot all attributes are bytestrings. Some are floats. Take a look at _FillValue, and valid_min and valid_max. To avoid an error, we use the isinstance function to check if the value of an attributes is type bytes - a bytestring. If it is, then we decode it. If not, we just extract the scalar and do nothing else.\nWe also fix the global attributes.\n\ndef fix_attributes(da):\n '''Decodes bytestring attributes to strings'''\n for attr, value in da.attrs.items():\n if isinstance(value[0], bytes):\n da.attrs[attr] = value[0].decode('utf-8')\n else:\n da.attrs[attr] = value[0]\n return\n\n# Fix variable attributes\nfor var in modis_ds.variables:\n da = modis_ds[var]\n fix_attributes(da)\n \n# Fix global attributes\nfix_attributes(modis_ds)\n\nWith this done, we can use the xarray function decode_cf to convert the attributes.\n\nmodis_ds = xr.decode_cf(modis_ds)\n\n\nmodis_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (nj: 2030, ni: 1354, time: 1)\nCoordinates:\n lat (nj, ni) float32 ...\n lon (nj, ni) float32 ...\n * time (time) datetime64[ns] 2019-06-22T10:00:01\nDimensions without coordinates: nj, ni\nData variables:\n sea_surface_temperature (time, nj, ni) float32 ...\n sst_dtime (time, nj, ni) timedelta64[ns] ...\n quality_level (time, nj, ni) float32 ...\n sses_bias (time, nj, ni) float32 ...\n sses_standard_deviation (time, nj, ni) float32 ...\n l2p_flags (time, nj, ni) int16 ...\n chlorophyll_a (time, nj, ni) float32 ...\n K_490 (time, nj, ni) float32 ...\n wind_speed (time, nj, ni) float32 ...\n dt_analysis (time, nj, ni) float32 ...\nAttributes: (12/49)\n Conventions: CF-1.7, ACDD-1.3\n title: MODIS Aqua L2P SST\n summary: Sea surface temperature retrievals produced a...\n references: GHRSST Data Processing Specification v2r5\n institution: NASA/JPL/OBPG/RSMAS\n history: MODIS L2P created at JPL PO.DAAC\n ... ...\n publisher_email: ghrsst-po@nceo.ac.uk\n processing_level: L2P\n cdm_data_type: swath\n startDirection: Ascending\n endDirection: Descending\n day_night_flag: Dayxarray.DatasetDimensions:nj: 2030ni: 1354time: 1Coordinates: (3)lat(nj, ni)float32...long_name :latitudestandard_name :latitudeunits :degrees_northvalid_min :-90.0valid_max :90.0comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[2748620 values with dtype=float32]lon(nj, ni)float32...long_name :longitudestandard_name :longitudeunits :degrees_eastvalid_min :-180.0valid_max :180.0comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[2748620 values with dtype=float32]time(time)datetime64[ns]2019-06-22T10:00:01long_name :reference time of sst filestandard_name :timecomment :time of first sensor observationcoverage_content_type :coordinatearray(['2019-06-22T10:00:01.000000000'], dtype='datetime64[ns]')Data variables: (10)sea_surface_temperature(time, nj, ni)float32...long_name :sea surface temperaturestandard_name :sea_surface_skin_temperatureunits :kelvinvalid_min :-1000valid_max :10000comment :sea surface temperature from thermal IR (11 um) channelssource :NASA and University of Miamicoverage_content_type :physicalMeasurement[2748620 values with dtype=float32]sst_dtime(time, nj, ni)timedelta64[ns]...long_name :time difference from reference timevalid_min :-32767valid_max :32767comment :time plus sst_dtime gives seconds after 00:00:00 UTC January 1, 1981coverage_content_type :referenceInformation[2748620 values with dtype=timedelta64[ns]]quality_level(time, nj, ni)float32...long_name :quality level of SST pixelvalid_min :0valid_max :5comment :thermal IR SST proximity confidence value; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valueflag_values :0flag_meanings :no_data bad_data worst_quality low_quality acceptable_quality best_qualitycoverage_content_type :qualityInformation[2748620 values with dtype=float32]sses_bias(time, nj, ni)float32...long_name :SSES bias error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :thermal IR SST bias error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]sses_standard_deviation(time, nj, ni)float32...long_name :SSES standard deviation error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :thermal IR SST standard deviation error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]l2p_flags(time, nj, ni)int16...long_name :L2P flagsvalid_min :0valid_max :16comment :These flags can be used to further filter data variablesflag_meanings :microwave land ice lake riverflag_masks :1coverage_content_type :qualityInformation[2748620 values with dtype=int16]chlorophyll_a(time, nj, ni)float32...long_name :Chlorophyll Concentration, OC3 Algorithmunits :mg m^-3valid_min :0.001valid_max :100.0comment :non L2P core fieldcoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]K_490(time, nj, ni)float32...long_name :Diffuse attenuation coefficient at 490 nm (OBPG)units :m^-1valid_min :50valid_max :30000comment :non L2P core fieldcoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]wind_speed(time, nj, ni)float32...long_name :10m wind speedstandard_name :wind_speedunits :m s-1valid_min :-127valid_max :127comment :Wind at 10 meters above the sea surfacesource :TBD. Placeholder. Currently emptygrid_mapping :TBDtime_offset :2.0height :10 mcoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]dt_analysis(time, nj, ni)float32...long_name :deviation from SST reference climatologyunits :kelvinvalid_min :-127valid_max :127comment :TBDsource :TBD. Placeholder. Currently emptycoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]Attributes: (49)Conventions :CF-1.7, ACDD-1.3title :MODIS Aqua L2P SSTsummary :Sea surface temperature retrievals produced at the NASA OBPG for the MODIS Aqua sensor. These have been reformatted to GHRSST GDS specifications by the JPL PO.DAACreferences :GHRSST Data Processing Specification v2r5institution :NASA/JPL/OBPG/RSMAShistory :MODIS L2P created at JPL PO.DAACcomment :L2P Core without DT analysis or other ancillary fields; Day, Start Node:Ascending, End Node:Descending; WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value; Refinedlicense :GHRSST and PO.DAAC protocol allow data use as free and open.id :MODIS_A-JPL-L2P-v2019.0naming_authority :org.ghrsstproduct_version :2019.0uuid :f6e1f61d-c4a4-4c17-8354-0c15e12d688bgds_version_id :2.0netcdf_version_id :4.1date_created :20200221T085224Zfile_quality_level :3spatial_resolution :1kmstart_time :20190622T100001Ztime_coverage_start :20190622T100001Zstop_time :20190622T100459Ztime_coverage_end :20190622T100459Znorthernmost_latitude :89.9862southernmost_latitude :66.2723easternmost_longitude :-45.9467westernmost_longitude :152.489source :MODIS sea surface temperature observations for the OBPGplatform :Aquasensor :MODISmetadata_link :http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MODIS_A-JPL-L2P-v2019.0keywords :Oceans > Ocean Temperature > Sea Surface Temperaturekeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordsstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventiongeospatial_lat_units :degrees_northgeospatial_lat_resolution :0.01geospatial_lon_units :degrees_eastgeospatial_lon_resolution :0.01acknowledgment :The MODIS L2P sea surface temperature data are sponsored by NASAcreator_name :Ed Armstrong, JPL PO.DAACcreator_email :edward.m.armstrong@jpl.nasa.govcreator_url :http://podaac.jpl.nasa.govproject :Group for High Resolution Sea Surface Temperaturepublisher_name :The GHRSST Project Officepublisher_url :http://www.ghrsst.orgpublisher_email :ghrsst-po@nceo.ac.ukprocessing_level :L2Pcdm_data_type :swathstartDirection :AscendingendDirection :Descendingday_night_flag :Day\n\n\nLet’s make a quick plot to take a look at the sea_surface_temperature variable.\n\nmodis_ds.sea_surface_temperature.plot() ;\n\n\n\n\n\n\nPlot MODIS and ICESat-2 data on a map\n\nmap_proj = ccrs.NorthPolarStereo()\n\nfig = plt.figure(figsize=(10,5))\nax = fig.add_subplot(projection=map_proj)\nax.coastlines()\n\n# Plot MODIS sst, save object as sst_img, so we can add colorbar\nsst_img = ax.pcolormesh(modis_ds.lon, modis_ds.lat, modis_ds.sea_surface_temperature[0,:,:], \n vmin=240, vmax=270, # Set max and min values for plotting\n cmap='viridis', shading='auto', # shading='auto' to avoid warning\n transform=ccrs.PlateCarree()) # coords are lat,lon but map if NPS \n\n# Plot IS2 surface height \nis2_img = ax.scatter(is2_ds.longitude, is2_ds.latitude,\n c=is2_ds.height_segment_height, \n vmax=1.5, # Set max height to plot\n cmap='Reds', alpha=0.6, s=2,\n transform=ccrs.PlateCarree())\n\n# Add colorbars\nfig.colorbar(sst_img, label='MODIS SST (K)')\nfig.colorbar(is2_img, label='ATL07 Height (m)')\n\n\n<matplotlib.colorbar.Colorbar at 0x7fd2eda24580>\n\n\n\n\n\n\n\nExtract SST coincident with ICESat-2 track\nThe MODIS SST is swath data, not a regularly-spaced grid of sea surface temperatures. ICESat-2 sea surface heights are irregularly spaced segments along one ground-track traced by the ATLAS instrument on-board ICESat-2. Fortunately, pyresample allows us to resample swath data.\npyresample has many resampling methods. We’re going to use the nearest neighbour resampling method, which is implemented using a k-dimensional tree algorithm or K-d tree. K-d trees are data structures that improve search efficiency for large data sets.\nThe first step is to define the geometry of the ICESat-2 and MODIS data. To do this we use the latitudes and longitudes of the datasets.\n\nis2_geometry = pyresample.SwathDefinition(lons=is2_ds.longitude,\n lats=is2_ds.latitude)\n\n\nmodis_geometry = pyresample.SwathDefinition(lons=modis_ds.lon, lats=modis_ds.lat)\n\nWe then implement the resampling method, passing the two geometries we have defined, the data array we want to resample - in this case sea surface temperature, and a search radius. The resampling method expects a numpy.Array rather than an xarray.DataArray, so we use values to get the data as a numpy.Array.\nWe set the search radius to 1000 m. The MODIS data is nominally 1km spacing.\n\nsearch_radius=1000.\nfill_value = np.nan\nis2_sst = pyresample.kd_tree.resample_nearest(\n modis_geometry,\n modis_ds.sea_surface_temperature.values,\n is2_geometry,\n search_radius,\n fill_value=fill_value\n)\n\n\nis2_sst\n\narray([263.375, 263.375, 263.375, ..., nan, nan, nan],\n dtype=float32)\n\n\n\nis2_ds['sea_surface_temperature'] = xr.DataArray(is2_sst, dims='segment')\nis2_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (segment: 235584)\nDimensions without coordinates: segment\nData variables:\n latitude (segment) float64 82.38 82.38 82.38 ... 72.61 72.61\n longitude (segment) float64 -55.11 -55.11 ... 145.1 145.1\n delta_time (segment) float64 4.642e+07 4.642e+07 ... 4.642e+07\n height_segment_height (segment) float32 nan nan nan ... -0.4335 -0.4463\n sea_surface_temperature (segment) float32 263.4 263.4 263.4 ... nan nan nanxarray.DatasetDimensions:segment: 235584Coordinates: (0)Data variables: (5)latitude(segment)float6482.38 82.38 82.38 ... 72.61 72.61contentType :referenceInformationcoordinates :delta_time longitudedescription :Latitude, WGS84, North=+, Lat of segment centerlong_name :Latitudesource :ATBD, section 4.4standard_name :latitudeunits :degrees_northvalid_max :90.0valid_min :-90.0array([82.38431982, 82.38431982, 82.38431982, ..., 72.60984638,\n 72.60977493, 72.60970985])longitude(segment)float64-55.11 -55.11 ... 145.1 145.1contentType :referenceInformationcoordinates :delta_time latitudedescription :Longitude, WGS84, East=+,Lon of segment centerlong_name :Longitudesource :ATBD, section 4.4standard_name :longitudeunits :degrees_eastvalid_max :180.0valid_min :-180.0array([-55.10896068, -55.10896068, -55.10896068, ..., 145.05396164,\n 145.05392851, 145.05389832])delta_time(segment)float644.642e+07 4.642e+07 ... 4.642e+07CLASS :DIMENSION_SCALENAME :gt1l/sea_ice_segments/delta_timeREFERENCE_LIST :[(<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)\n (<HDF5 object reference>, 0) (<HDF5 object reference>, 0)]contentType :physicalMeasurementcoordinates :latitude longitudedescription :Number of GPS seconds since the ATLAS SDP epoch. The ATLAS Standard Data Products (SDP) epoch offset is defined within /ancillary_data/atlas_sdp_gps_epoch as the number of GPS seconds between the GPS epoch (1980-01-06T00:00:00.000000Z UTC) and the ATLAS SDP epoch. By adding the offset contained within atlas_sdp_gps_epoch to delta time parameters, the time in gps_seconds relative to the GPS epoch can be computed.long_name :Elapsed GPS secondssource :telemetrystandard_name :timeunits :seconds since 2018-01-01array([46419293.64266939, 46419293.64266939, 46419293.64266939, ...,\n 46419681.87646231, 46419681.87759533, 46419681.87862704])height_segment_height(segment)float32nan nan nan ... -0.4335 -0.4463_FillValue :3.4028235e+38contentType :referenceInformationcoordinates :../delta_time ../latitude ../longitudedescription :Mean height from along-track segment fit detremined by the sea ice algorithm. The sea ice height is relative to the tide-free MSS.long_name :height of segment surfacesource :ATBD, section 4.2.2.4units :metersarray([ nan, nan, nan, ..., -0.46550068,\n -0.43347716, -0.4462675 ], dtype=float32)sea_surface_temperature(segment)float32263.4 263.4 263.4 ... nan nan nanarray([263.375, 263.375, 263.375, ..., nan, nan, nan],\n dtype=float32)Attributes: (0)\n\n\n\n\nPlot SST and Height along track\nThis is a quick plot of the extracted data. We’re using matplotlib so we can use latitude as the x-value:\n\nis2_ds = is2_ds.set_coords(['latitude'])\n\nfig, ax1 = plt.subplots(figsize=(15, 7))\nax1.set_xlim(82.,88.)\nax1.plot(is2_ds.latitude, is2_ds.sea_surface_temperature, \n color='orange', label='SST', zorder=3)\nax1.set_ylabel('SST (K)')\n\nax2 = ax1.twinx()\nax2.plot(is2_ds.latitude, is2_ds.height_segment_height, label='Height')\nax2.set_ylabel('Height (m)')\n\nfig.legend()\n\n<matplotlib.legend.Legend at 0x7fd2ef2ea880>" + "objectID": "external/appeears_csv_cloud_access.html#contact-info", + "href": "external/appeears_csv_cloud_access.html#contact-info", + "title": "How to work with AppEEARS Point Sample CSV outputs", + "section": "Contact Info:", + "text": "Contact Info:\nEmail: LPDAAC@usgs.gov\nVoice: +1-866-573-3222\nOrganization: Land Processes Distributed Active Archive Center (LP DAAC)¹\nWebsite: https://lpdaac.usgs.gov/\nDate last modified: 05-06-2023\n¹Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I." }, { "objectID": "workshops/index.html", @@ -3239,5 +3057,138 @@ "title": "Workshops & Hackathons", "section": "2021 Cloud Hackathon", "text": "2021 Cloud Hackathon\nhttps://nasa-openscapes.github.io/2021-Cloud-Hackathon\nThe Cloud Hackathon: Transitioning Earthdata Workflows to the Cloud is a virtual 5-day (4 hours per day) collaborative open science learning experience aimed at exploring, creating, and promoting effective cloud-based science and applications workflows using NASA Earthdata Cloud data, tools, and services (among others), in support of Earth science data processing and analysis in the era of big data. Its goals are to:\n\nIntroduce Earth science data users to NASA Earthdata cloud-based data products, tools and services in order to increase awareness and support transition to cloud-based science and applications workflows.\nEnable science and applications workflows in the cloud that leverage NASA Earth Observations and capabilities (services) from within the NASA Earthdata Cloud, hosted in Amazon Web Services (AWS) cloud, thus increasing NASA Earthdata data utility and meaningfulness for science and applications use cases.\nFoster community engagement utilizing Earthdata cloud tools and services in support of open science and open data." + }, + { + "objectID": "examples/ORNL/ORNL_DAYMET__Kerchunk.html", + "href": "examples/ORNL/ORNL_DAYMET__Kerchunk.html", + "title": "EarthData Cloud Cookbook", + "section": "", + "text": "!mamba install kerchunk -y --quiet\n\n\n!mamba install h5py=3.2 -y --quiet\n\n\nimport numpy as np\nimport xarray as xr\nimport matplotlib.pyplot as plt\nimport s3fs\nimport datetime as dt\nimport logging\nimport fsspec\nimport ujson\nimport requests\nfrom tqdm import tqdm\nfrom glob import glob\n\n\nimport kerchunk.hdf\n\n\ns3_cred_endpoint = {\n 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n 'ornldaac':'https://data.ornldaac.earthdata.nasa.gov/s3credentials'\n}\n\n\ndef get_temp_creds():\n temp_creds_url = s3_cred_endpoint['ornldaac']\n return requests.get(temp_creds_url).json()\n\n\ntemp_creds_req = get_temp_creds()\n\n\n#fs = fsspec.filesystem('s3', anon=True)\nfs = s3fs.S3FileSystem(anon=False, key=temp_creds_req['accessKeyId'], secret=temp_creds_req['secretAccessKey'], token=temp_creds_req['sessionToken'])\n\n\nurls = ['s3://ornl-cumulus-prod-protected/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_tmax_2015.nc',\n 's3://ornl-cumulus-prod-protected/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_tmax_2016.nc',\n 's3://ornl-cumulus-prod-protected/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_tmax_2017.nc',\n 's3://ornl-cumulus-prod-protected/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_tmax_2018.nc',\n 's3://ornl-cumulus-prod-protected/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_tmax_2019.nc',\n 's3://ornl-cumulus-prod-protected/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_tmax_2020.nc']\n\n\ndef gen_json(u):\n so = dict(\n mode=\"rb\", anon=False, default_fill_cache=False,\n default_cache_type=\"none\"\n )\n with fs.open(u, **so) as infile:\n print(u)\n h5chunks = kerchunk.hdf.SingleHdf5ToZarr(infile, u, inline_threshold=300)\n with open(f\"jsons/{u.split('/')[-1]}.json\", 'wb') as outfile:\n outfile.write(ujson.dumps(h5chunks.translate()).encode())\n\n\nfor u in urls:\n gen_json(u)\n\n\nu = 's3://ornl-cumulus-prod-protected/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_tmax_2015.nc'\n\n\nso = dict(\n mode=\"rb\", anon=False, default_fill_cache=False, default_cache_type=\"none\"\n )\n\n\ninf = fs.open(u, **so)\n\n\nh5chunks = kerchunk.hdf.SingleHdf5ToZarr(inf, u, inline_threshold=300)\n\n\nh5chunks.translate()\n\nPermissionError: The provided token has expired." + }, + { + "objectID": "examples/index.html", + "href": "examples/index.html", + "title": "Example Workflows", + "section": "", + "text": "This section contains end-to-end example workflows for processing and analyzing data in the cloud.\nE.g. Here are some examples for Kerchunk!" + }, + { + "objectID": "examples/NSIDC/ICESat2-CMR-AWS-S3.html", + "href": "examples/NSIDC/ICESat2-CMR-AWS-S3.html", + "title": "Accessing NSIDC Cloud Collections Using CMR", + "section": "", + "text": "Programmatic access of NSIDC data can happen in 2 ways:" + }, + { + "objectID": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#the-big-picture", + "href": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#the-big-picture", + "title": "Accessing NSIDC Cloud Collections Using CMR", + "section": "The big picture:", + "text": "The big picture:\nThere is nothing wrong with downloading data to our local machine but that can get complicated or even impossible if a dataset is too large. For this reason NSIDC along with other NASA data centers started to collocate or migrate their dataset holdings to the cloud.\n\nSteps\n\nAuthenticate with the NASA Earthdata Login API (EDL).\nSearch granules/collections using a CMR client that supports authentication\nParse CMR responses and get AWS S3 URLs\nAccess the data granules using temporary AWS credentials given by the NSIDC cloud credentials endpoint\n\n\n\nData used:\n\nICESat-2 ATL03: This data set contains height above the WGS 84 ellipsoid (ITRF2014 reference frame), latitude, longitude, and time for all photons.\n\n\n\nRequirements\n\nNASA Eartdata Login (EDL) credentials\npython libraries:\n\nh5py\nmatplotlib\nxarray\ns3fs\npython-cmr\n\ncmr helpers: included in this notebook" + }, + { + "objectID": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#querying-cmr-for-nsidc-data-in-the-cloud", + "href": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#querying-cmr-for-nsidc-data-in-the-cloud", + "title": "Accessing NSIDC Cloud Collections Using CMR", + "section": "Querying CMR for NSIDC data in the cloud", + "text": "Querying CMR for NSIDC data in the cloud\nMost collections at NSIDC have not being migrated to the cloud and can be found using CMR with no authentication at all. Here is a simple example for altimeter data (ATL03) coming from the ICESat-2 mission. First we’ll search the regular collection and then we’ll do the same using the cloud collection.\nNote: This notebook uses CMR to search and locate the data granules, this is not the only workflow for data access and discovery.\n\nHarmonyPy: Uses Harmony the NASA API to search, subset and transform the data in the cloud.\ncmr-stac: A “static” metadata catalog than can be read by Intake oand other client libraries to optimize the access of files in the cloud." + }, + { + "objectID": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#cloud-collections", + "href": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#cloud-collections", + "title": "Accessing NSIDC Cloud Collections Using CMR", + "section": "Cloud Collections", + "text": "Cloud Collections\nSome NSIDC cloud collections are not yet, which means that temporarily you’ll have to request access emailing nsidc@nsidc.org so your Eartdata login is in the authorized list for early users.\n\nimport getpass\nimport textwrap\n\nfrom cmr.search import collection as cmr_collection\nfrom cmr.search import granule \nfrom cmr.auth import token\n\nfrom cmr_auth import CMRAuth\n\n# NON_AWS collections are hosted at the NSIDC DAAC data center\n# AWS_CLOUD collections are hosted at AWS S3 us-west-2\nNSIDC_PROVIDERS = {\n 'NSIDC_HOSTED': 'NSIDC_ECS', \n 'AWS_HOSTED':'NSIDC_CPRD'\n}\n\n# Use your own EDL username\nUSER = 'betolink'\n\nprint('Enter your NASA Earthdata login password:')\npassword = getpass.getpass()\n# This helper class will handle credentials with CMR\nCMR_auth = CMRAuth(USER, password)\n# Token to search preliminary collections on CMR\ncmr_token = CMR_auth.get_token()\n\n\n# The query object uses a simple python dictionary\nquery = {'short_name':'ATL03',\n 'token': cmr_token,\n 'provider': NSIDC_PROVIDERS['AWS_HOSTED']}\n\ncollections = cmr_collection.search(query)\n\nfor collection in collections[0:3]:\n wrapped_abstract = '\\n'.join(textwrap.wrap(f\"Abstract: {collection['umm']['Abstract']}\", 80)) + '\\n'\n print(f\"concept-id: {collection['meta']['concept-id']}\\n\" +\n f\"Title: {collection['umm']['EntryTitle']}\\n\" +\n wrapped_abstract)" + }, + { + "objectID": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#searching-for-data-granules-in-the-cloud-with-cmr", + "href": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#searching-for-data-granules-in-the-cloud-with-cmr", + "title": "Accessing NSIDC Cloud Collections Using CMR", + "section": "Searching for data granules in the cloud with CMR", + "text": "Searching for data granules in the cloud with CMR\nCMR uses different collection id’s for datasets in the cloud.\n\n# now that we have the concept-id for our ATL03 in the cloud we do the same thing we did with ATL03 hosted at\nfrom cmr_serializer import QueryResult\n# NSIDC but using the cloud concept-id\n# Jeneau ice sheet\nquery = {'concept-id': 'C2027878642-NSIDC_CPRD',\n 'token': cmr_token,\n 'bounding_box': '-135.1977,58.3325,-133.3410,58.9839'}\n\n# Querying for ATL03 v3 using its concept-id and a bounding box\nresults = granule.search(query, limit=1000)\ngranules = QueryResult(results).items()\n\nprint(f\"Total granules found: {len(results)} \\n\")\n\n# Print the first 3 granules\nfor g in granules[0:3]:\n display(g)\n # You can use: print(g) for the regular text representation.\n\n\nNOTE: Not all the data granules for NSIDC datasets have been migrated to S3. This might result in different counts between the NSIDC hosted data collections and the ones in AWS S3\n\n\n# We can list the s3 links but \nfor g in granules:\n for link in g.data_links():\n print(link)\n\nWe note that our RelatedLinks array now contain links to AWS S3, these are the direct URIs for our data granules in the AWS us-west-2 region." + }, + { + "objectID": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#data-access-using-aws-s3", + "href": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#data-access-using-aws-s3", + "title": "Accessing NSIDC Cloud Collections Using CMR", + "section": "Data Access using AWS S3", + "text": "Data Access using AWS S3\n\nIMPORTANT: This section will only work if this notebook is running on the AWS us-west-2 zone\n\nThere is more than one way of accessing data on AWS S3, either downloading it to your local machine using the official client library or using a python library.\nPerformance tip: using the HTTPS URLs will decrease the access performance since these links have to internally be processed by AWS’s content delivery system (CloudFront). To get a better performance we should access the S3:// URLs with BOTO3 or a high level S3 enabled library (i.e. S3FS)\nRelated links: * HDF in the Cloud challenges and solutions for scientific data * Cloud Storage (Amazon S3) HDF5 Connector\n\nimport s3fs\nimport h5py\n\nimport xarray as xr\nimport numpy as np\n\nimport matplotlib.pyplot as plt\nimport cartopy.crs as ccrs\nimport cartopy.feature as cfeature\n\n\n# READ only temporary credentials\n# This credentials only last 1 hour.\ns3_cred = CMR_auth.get_s3_credentials()\n\ns3_fs = s3fs.S3FileSystem(key=s3_cred['accessKeyId'],\n secret=s3_cred['secretAccessKey'],\n token=s3_cred['sessionToken'])\n\n# Now you could grab S3 links to your cloud instance (EC2, Hub etc) using:\n# s3_fs.get('s3://SOME_LOCATION/ATL03_20181015124359_02580106_004_01.h5', 'test.h5')\n\nNow that we have the propper credentials in our file mapper, we can access the data within AWS us-west-2.\nIf we are not running this notebook in us-west-2 will get an access denied error" + }, + { + "objectID": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#using-xarray-to-open-files-on-s3", + "href": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#using-xarray-to-open-files-on-s3", + "title": "Accessing NSIDC Cloud Collections Using CMR", + "section": "Using xarray to open files on S3", + "text": "Using xarray to open files on S3\nATL data is complex so xarray doesn’t know how to extract the important bits out of it.\n\nwith s3_fs.open('s3://nsidc-cumulus-prod-protected/ATLAS/ATL03/004/2018/10/15/ATL03_20181015124359_02580106_004_01.h5', 'rb') as s3f:\n with h5py.File(s3f, 'r') as f:\n print([key for key in f.keys()])\n gt1l = xr.Dataset({'height': (['x'], f['gt1l']['heights']['h_ph'][:]),\n 'latitude': (['x'], f['gt1l']['heights']['lat_ph'][:]), \n 'longitude': (['x'], f['gt1l']['heights']['lon_ph'][:]),\n 'dist_ph': (['x'], f['gt1l']['heights']['dist_ph_along'][:])})\ngt1l" + }, + { + "objectID": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#plotting-the-data", + "href": "examples/NSIDC/ICESat2-CMR-AWS-S3.html#plotting-the-data", + "title": "Accessing NSIDC Cloud Collections Using CMR", + "section": "Plotting the data", + "text": "Plotting the data\n\ngt1l.height.plot()\n\n\n%matplotlib widget\n\nfig, ax = plt.subplots(figsize=(14, 4))\n\ngt1l.height.plot(ax=ax, ls='', marker='o', ms=1)\ngt1l.height.rolling(x=1000, min_periods=500, center=True).mean().plot(ax=ax, c='k', lw=2)\nax.set_xlabel('Along track distance (m)', fontsize=12);\nax.set_ylabel('Photon Height (m)', fontsize=12)\nax.set_title('ICESat-2 ATL03', fontsize=14)\nax.tick_params(axis='both', which='major', labelsize=12)\n\nsubax = fig.add_axes([0.69,0.50,0.3,0.3], projection=ccrs.NorthPolarStereo())\nsubax.set_aspect('equal')\nsubax.set_extent([-180., 180., 30, 90.], ccrs.PlateCarree())\nsubax.add_feature(cfeature.LAND)\nsubax.plot(gt1l.longitude, gt1l.latitude, transform=ccrs.PlateCarree(), lw=1);\n\nfig.savefig('test.png')\n\nplt.show()" + }, + { + "objectID": "examples/Transform-workflow.html", + "href": "examples/Transform-workflow.html", + "title": "Earthdata Cloud Exploration: Direct Access and Harmony workflow", + "section": "", + "text": "This notebook will walk through two different access and transformation options across a single dataset available in the Earthdata Cloud: Direct s3 access and Harmony transformation services. These steps are based off of several notebook tutorials presented during the 2021 Earthdata Cloud Hackathon and 2021 AGU Workshop.\nThe following use case focuses on Sea Surface Temperature (SST) from the GHRSST MODIS L2 dataset (MODIS_T-JPL-L2P-v2019.0) available from PO.DAAC in the Earthdata Cloud.\n\n\nUsing NASA Worldview, we can first explore SST during a tropical storm event; in this case, we can overlay L2 and L4 SST variables against true color imagery to observe Hurricane Ida in August 2021. Although this is a small sample set, this use case could be expanded to explore how SST responds during the Atlantic hurricane over the next several months. The same data that we are requesting below using direct s3 access and via Harmony-py can also be requested using NASA Earthdata Search\n\n\n\nHurrican Ida snapshot - Worldview\n\n\nWe will access a single netCDF file from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataset. This approach leverages S3 native protocols for efficient access to the data.\nAfter inspecting the data, we will then access the same file, but in this case we will also subset the data to our area of interest using Harmony." + }, + { + "objectID": "examples/Transform-workflow.html#summary", + "href": "examples/Transform-workflow.html#summary", + "title": "Earthdata Cloud Exploration: Direct Access and Harmony workflow", + "section": "", + "text": "This notebook will walk through two different access and transformation options across a single dataset available in the Earthdata Cloud: Direct s3 access and Harmony transformation services. These steps are based off of several notebook tutorials presented during the 2021 Earthdata Cloud Hackathon and 2021 AGU Workshop.\nThe following use case focuses on Sea Surface Temperature (SST) from the GHRSST MODIS L2 dataset (MODIS_T-JPL-L2P-v2019.0) available from PO.DAAC in the Earthdata Cloud.\n\n\nUsing NASA Worldview, we can first explore SST during a tropical storm event; in this case, we can overlay L2 and L4 SST variables against true color imagery to observe Hurricane Ida in August 2021. Although this is a small sample set, this use case could be expanded to explore how SST responds during the Atlantic hurricane over the next several months. The same data that we are requesting below using direct s3 access and via Harmony-py can also be requested using NASA Earthdata Search\n\n\n\nHurrican Ida snapshot - Worldview\n\n\nWe will access a single netCDF file from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataset. This approach leverages S3 native protocols for efficient access to the data.\nAfter inspecting the data, we will then access the same file, but in this case we will also subset the data to our area of interest using Harmony." + }, + { + "objectID": "examples/Transform-workflow.html#import-required-packages", + "href": "examples/Transform-workflow.html#import-required-packages", + "title": "Earthdata Cloud Exploration: Direct Access and Harmony workflow", + "section": "Import Required Packages", + "text": "Import Required Packages\n\n# Earthdata Login\nfrom netrc import netrc\nfrom subprocess import Popen\nfrom platform import system\nfrom getpass import getpass\nimport os\n\n# Direct access\nimport requests\nimport s3fs\nimport xarray as xr\nimport hvplot.xarray\n\n# Harmony\nfrom harmony import BBox, Client, Collection, Request, LinkType\nfrom harmony.config import Environment\nfrom pprint import pprint\nimport datetime as dt" + }, + { + "objectID": "examples/Transform-workflow.html#authentication-for-nasa-earthdata", + "href": "examples/Transform-workflow.html#authentication-for-nasa-earthdata", + "title": "Earthdata Cloud Exploration: Direct Access and Harmony workflow", + "section": "1. Authentication for NASA Earthdata", + "text": "1. Authentication for NASA Earthdata\nBased off of https://nasa-openscapes.github.io/2021-Cloud-Workshop-AGU/tutorials/02_NASA_Earthdata_Authentication.html\nFirst, we will create a hidden .netrc file (_netrc for Window OS) with Earthdata login credentials in your home directory. This file is needed to access NASA Earthdata assets from a scripting environment like Python.\n\nEarthdata Login\nAn Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n\n\nAuthentication via netrc File\nYou will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. An example of the required content is below.\nmachine urs.earthdata.nasa.gov\nlogin <USERNAME>\npassword <PASSWORD>\n<USERNAME> and <PASSWORD> would be replaced by your actual Earthdata Login username and password respectively.\nThe code below will:\n\ncheck what operating system (OS) is being used to determine which netrc file to check for/create (.netrc or _netrc)\ncheck if you have an netrc file, and if so, varify if those credentials are for the Earthdata endpoint\ncreate a netrc file if a netrc file is not present.\n\n\nurs = 'urs.earthdata.nasa.gov' # Earthdata URL endpoint for authentication\nprompts = ['Enter NASA Earthdata Login Username: ',\n 'Enter NASA Earthdata Login Password: ']\n\n# Determine the OS (Windows machines usually use an '_netrc' file)\nnetrc_name = \"_netrc\" if system()==\"Windows\" else \".netrc\"\n\n# Determine if netrc file exists, and if so, if it includes NASA Earthdata Login Credentials\ntry:\n netrcDir = os.path.expanduser(f\"~/{netrc_name}\")\n netrc(netrcDir).authenticators(urs)[0]\n\n# Below, create a netrc file and prompt user for NASA Earthdata Login Username and Password\nexcept FileNotFoundError:\n homeDir = os.path.expanduser(\"~\")\n Popen('touch {0}{2} | echo machine {1} >> {0}{2}'.format(homeDir + os.sep, urs, netrc_name), shell=True)\n Popen('echo login {} >> {}{}'.format(getpass(prompt=prompts[0]), homeDir + os.sep, netrc_name), shell=True)\n Popen('echo \\'password {} \\'>> {}{}'.format(getpass(prompt=prompts[1]), homeDir + os.sep, netrc_name), shell=True)\n # Set restrictive permissions\n Popen('chmod 0600 {0}{1}'.format(homeDir + os.sep, netrc_name), shell=True)\n\n # Determine OS and edit netrc file if it exists but is not set up for NASA Earthdata Login\nexcept TypeError:\n homeDir = os.path.expanduser(\"~\")\n Popen('echo machine {1} >> {0}{2}'.format(homeDir + os.sep, urs, netrc_name), shell=True)\n Popen('echo login {} >> {}{}'.format(getpass(prompt=prompts[0]), homeDir + os.sep, netrc_name), shell=True)\n Popen('echo \\'password {} \\'>> {}{}'.format(getpass(prompt=prompts[1]), homeDir + os.sep, netrc_name), shell=True)\n\n\nSee if the file was created\nIf the file was created, we’ll see a .netrc file (_netrc for Window OS) in the list printed below. To view the contents from a Jupyter environment, click File on the top toolbar, select Open from Path…, type .netrc, and click Open. The .netrc file will open within the text editor.\n\n!!! Beware, your password will be visible if the .netrc file is opened in the text editor.\n\n\n!ls -al ~/" + }, + { + "objectID": "examples/Transform-workflow.html#accessing-a-netcdf4hdf5-file---s3-direct-access", + "href": "examples/Transform-workflow.html#accessing-a-netcdf4hdf5-file---s3-direct-access", + "title": "Earthdata Cloud Exploration: Direct Access and Harmony workflow", + "section": "2. Accessing a NetCDF4/HDF5 File - S3 Direct Access", + "text": "2. Accessing a NetCDF4/HDF5 File - S3 Direct Access\n_Based off of https://nasa-openscapes.github.io/2021-Cloud-Workshop-AGU/how-tos/Earthdata_Cloud__Single_File_Direct_S3_Access_NetCDF4_Example.html and https://nasa-openscapes.github.io/2021-Cloud-Workshop-AGU/tutorials/04_On-Prem_Cloud.html\n\nSummary\nWe will access a single netCDF file from the GHRSST MODIS L2 dataset (MODIS_T-JPL-L2P-v2019.0) inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataset. This approach leverages S3 native protocols for efficient access to the data.\n\n\nRequirements\n\n1. AWS instance running in us-west-2\nNASA Earthdata Cloud data in S3 can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region.\n\n\n2. Earthdata Login\n\n\n3. netrc File\n\n\n\nLearning Objectives\n\nhow to retrieve temporary S3 credentials for in-region direct S3 bucket access\nhow to define a dataset of interest and find netCDF files in S3 bucket\nhow to perform in-region direct access of MODIS_T-JPL-L2P-v2019.0 data in S3\nhow to plot the data\n\n\n\nGet Temporary AWS Credentials\nDirect S3 access is achieved by passing NASA supplied temporary credentials to AWS so we can interact with S3 objects from applicable Earthdata Cloud buckets. For now, each NASA DAAC has different AWS credentials endpoints. Below are some of the credential endpoints to various DAACs:\n\ns3_cred_endpoint = {\n 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n 'gesdisc': 'https://data.gesdisc.earthdata.nasa.gov/s3credentials',\n 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n 'ornldaac': 'https://data.ornldaac.earthdata.nasa.gov/s3credentials',\n 'ghrcdaac': 'https://data.ghrc.earthdata.nasa.gov/s3credentials'\n}\n\nCreate a function to make a request to an endpoint for temporary credentials. Remember, each DAAC has their own endpoint and credentials are not usable for cloud data from other DAACs.\n\ndef get_temp_creds(provider):\n return requests.get(s3_cred_endpoint[provider]).json()\n\n\ntemp_creds_req = get_temp_creds('podaac')\n#temp_creds_req\n\n\n\nSet up an s3fs session for Direct Access\ns3fs sessions are used for authenticated access to s3 buckets and allow for typical file-system style operations. Below we create session by passing in the temporary credentials we recieved from our temporary credentials endpoint.\n\nfs_s3 = s3fs.S3FileSystem(anon=False, \n key=temp_creds_req['accessKeyId'], \n secret=temp_creds_req['secretAccessKey'], \n token=temp_creds_req['sessionToken'],\n client_kwargs={'region_name':'us-west-2'})\n\nIn this example we’re interested in the GHRSST MODIS L2 collection from NASA’s PO.DAAC in Earthdata Cloud. Below we specify the s3 URL to the data asset in Earthdata Cloud. This URL can be found via Earthdata Search or programmatically through the CMR and CMR-STAC APIs.\n\nshort_name = 'MODIS_T-JPL-L2P-v2019.0'\ns3_url = 's3://podaac-ops-cumulus-protected/MODIS_T-JPL-L2P-v2019.0/20210820033500-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0.nc'\n\n\n\nDirect In-region Access\nOpen with the netCDF file using the s3fs package, then load the cloud asset into a concatenated xarray dataset.\n\ns3_file_obj = fs_s3.open(s3_url, mode='rb')\n\n\nssh_ds = xr.open_dataset(s3_file_obj)\nssh_ds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (nj: 2030, ni: 1354, time: 1)\nCoordinates:\n lat (nj, ni) float32 ...\n lon (nj, ni) float32 ...\n * time (time) datetime64[ns] 2021-08-20T03:35:00\nDimensions without coordinates: nj, ni\nData variables:\n sea_surface_temperature (time, nj, ni) float32 ...\n sst_dtime (time, nj, ni) timedelta64[ns] ...\n quality_level (time, nj, ni) float32 ...\n sses_bias (time, nj, ni) float32 ...\n sses_standard_deviation (time, nj, ni) float32 ...\n l2p_flags (time, nj, ni) int16 ...\n sea_surface_temperature_4um (time, nj, ni) float32 ...\n quality_level_4um (time, nj, ni) float32 ...\n sses_bias_4um (time, nj, ni) float32 ...\n sses_standard_deviation_4um (time, nj, ni) float32 ...\n wind_speed (time, nj, ni) float32 ...\n dt_analysis (time, nj, ni) float32 ...\nAttributes: (12/49)\n Conventions: CF-1.7, ACDD-1.3\n title: MODIS Terra L2P SST\n summary: Sea surface temperature retrievals produced a...\n references: GHRSST Data Processing Specification v2r5\n institution: NASA/JPL/OBPG/RSMAS\n history: MODIS L2P created at JPL PO.DAAC\n ... ...\n publisher_email: ghrsst-po@nceo.ac.uk\n processing_level: L2P\n cdm_data_type: swath\n startDirection: Ascending\n endDirection: Ascending\n day_night_flag: Nightxarray.DatasetDimensions:nj: 2030ni: 1354time: 1Coordinates: (3)lat(nj, ni)float32...long_name :latitudestandard_name :latitudeunits :degrees_northvalid_min :-90.0valid_max :90.0comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[2748620 values with dtype=float32]lon(nj, ni)float32...long_name :longitudestandard_name :longitudeunits :degrees_eastvalid_min :-180.0valid_max :180.0comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[2748620 values with dtype=float32]time(time)datetime64[ns]2021-08-20T03:35:00long_name :reference time of sst filestandard_name :timecomment :time of first sensor observationcoverage_content_type :coordinatearray(['2021-08-20T03:35:00.000000000'], dtype='datetime64[ns]')Data variables: (12)sea_surface_temperature(time, nj, ni)float32...long_name :sea surface temperaturestandard_name :sea_surface_skin_temperatureunits :kelvinvalid_min :-1000valid_max :10000comment :sea surface temperature from thermal IR (11 um) channelssource :NASA and University of Miamicoverage_content_type :physicalMeasurement[2748620 values with dtype=float32]sst_dtime(time, nj, ni)timedelta64[ns]...long_name :time difference from reference timevalid_min :-32767valid_max :32767comment :time plus sst_dtime gives seconds after 00:00:00 UTC January 1, 1981coverage_content_type :referenceInformation[2748620 values with dtype=timedelta64[ns]]quality_level(time, nj, ni)float32...long_name :quality level of SST pixelvalid_min :0valid_max :5comment :thermal IR SST proximity confidence value; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valueflag_values :[0 1 2 3 4 5]flag_meanings :no_data bad_data worst_quality low_quality acceptable_quality best_qualitycoverage_content_type :qualityInformation[2748620 values with dtype=float32]sses_bias(time, nj, ni)float32...long_name :SSES bias error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :thermal IR SST bias error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]sses_standard_deviation(time, nj, ni)float32...long_name :SSES standard deviation error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :thermal IR SST standard deviation error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]l2p_flags(time, nj, ni)int16...long_name :L2P flagsvalid_min :0valid_max :16comment :These flags can be used to further filter data variablesflag_meanings :microwave land ice lake riverflag_masks :[ 1 2 4 8 16]coverage_content_type :qualityInformation[2748620 values with dtype=int16]sea_surface_temperature_4um(time, nj, ni)float32...long_name :sea surface temperatureunits :kelvinvalid_min :-1000valid_max :10000comment :sea surface temperature from mid-IR (4 um) channels; non L2P core fieldcoverage_content_type :physicalMeasurement[2748620 values with dtype=float32]quality_level_4um(time, nj, ni)float32...long_name :quality level of SST pixelvalid_min :0valid_max :5comment :mid-IR SST proximity confidence value; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valueflag_values :[0 1 2 3 4 5]flag_meanings :no_data bad_data worst_quality low_quality acceptable_quality best_qualitycoverage_content_type :qualityInformation[2748620 values with dtype=float32]sses_bias_4um(time, nj, ni)float32...long_name :SSES bias error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :mid-IR SST bias error; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]sses_standard_deviation_4um(time, nj, ni)float32...long_name :SSES standard deviation error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :mid-IR SST standard deviation error; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]wind_speed(time, nj, ni)float32...long_name :10m wind speedstandard_name :wind_speedunits :m s-1valid_min :-127valid_max :127comment :Wind at 10 meters above the sea surfacesource :TBD. Placeholder. Currently emptygrid_mapping :TBDtime_offset :2.0height :10 mcoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]dt_analysis(time, nj, ni)float32...long_name :deviation from SST reference climatologyunits :kelvinvalid_min :-127valid_max :127comment :TBDsource :TBD. Placeholder. Currently emptycoverage_content_type :auxiliaryInformation[2748620 values with dtype=float32]Attributes: (49)Conventions :CF-1.7, ACDD-1.3title :MODIS Terra L2P SSTsummary :Sea surface temperature retrievals produced at the NASA OBPG for the MODIS Aqua sensor. These have been reformatted to GHRSST GDS specifications by the JPL PO.DAACreferences :GHRSST Data Processing Specification v2r5institution :NASA/JPL/OBPG/RSMAShistory :MODIS L2P created at JPL PO.DAACcomment :L2P Core without DT analysis or other ancillary fields; Night, Start Node:Ascending, End Node:Ascending; WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value; Quicklooklicense :GHRSST and PO.DAAC protocol allow data use as free and open.id :MODIS_T-JPL-L2P-v2019.0naming_authority :org.ghrsstproduct_version :2019.0uuid :f6e1f61d-c4a4-4c17-8354-0c15e12d688bgds_version_id :2.0netcdf_version_id :4.1date_created :20210820T153042Zfile_quality_level :3spatial_resolution :1kmstart_time :20210820T033500Ztime_coverage_start :20210820T033500Zstop_time :20210820T033959Ztime_coverage_end :20210820T033959Znorthernmost_latitude :34.0296southernmost_latitude :12.9895easternmost_longitude :-68.4372westernmost_longitude :-95.5518source :MODIS sea surface temperature observations for the OBPGplatform :Terrasensor :MODISmetadata_link :http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MODIS_T-JPL-L2P-v2019.0keywords :Oceans > Ocean Temperature > Sea Surface Temperaturekeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordsstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventiongeospatial_lat_units :degrees_northgeospatial_lat_resolution :0.01geospatial_lon_units :degrees_eastgeospatial_lon_resolution :0.01acknowledgment :The MODIS L2P sea surface temperature data are sponsored by NASAcreator_name :Ed Armstrong, JPL PO.DAACcreator_email :edward.m.armstrong@jpl.nasa.govcreator_url :http://podaac.jpl.nasa.govproject :Group for High Resolution Sea Surface Temperaturepublisher_name :The GHRSST Project Officepublisher_url :http://www.ghrsst.orgpublisher_email :ghrsst-po@nceo.ac.ukprocessing_level :L2Pcdm_data_type :swathstartDirection :AscendingendDirection :Ascendingday_night_flag :Night\n\n\nLet’s make a quick plot to take a look at the sea_surface_temperature variable.\n\nssh_ds.sea_surface_temperature.plot() ;" + }, + { + "objectID": "examples/Transform-workflow.html#accessing-multiple-netcdf4hdf5-files---data-subsetting-and-transformation-services-in-the-cloud", + "href": "examples/Transform-workflow.html#accessing-multiple-netcdf4hdf5-files---data-subsetting-and-transformation-services-in-the-cloud", + "title": "Earthdata Cloud Exploration: Direct Access and Harmony workflow", + "section": "3. Accessing Multiple NetCDF4/HDF5 Files - Data Subsetting and Transformation Services in the Cloud", + "text": "3. Accessing Multiple NetCDF4/HDF5 Files - Data Subsetting and Transformation Services in the Cloud\n\nUsing the Harmony-Py library to access customized data from NASA Earthdata\n\nWhat other access options or services exist for this dataset?\nMaybe we’re interested in creating a time series over a larger area or with a larger dataset. Let’s see whether there are other services out there that could either make this operation more performant in the cloud (with a cloud-optimized output) or subsetting services to reduce the data volume.\nBased off of https://nasa-openscapes.github.io/2021-Cloud-Hackathon/tutorials/07_Harmony_Subsetting.html\n\n\n\nSummary\nWe have already explored direct access to the NASA EOSDIS archive in the cloud via the Amazon Simple Storage Service (S3) by using the Common Metadata Repository (CMR) to search for granule locations. In addition to directly accessing the files archived and distributed by each of the NASA DAACs, many datasets also support services that allow us to customize the data via subsetting, reformatting, reprojection, and other transformations.\nThis tutorial demonstrates how to find, request, and use customized data from a new ecosystem of services operating within the NASA Earthdata Cloud: NASA Harmony.\n\n\nBenefits\nBut first, why use this option when we’ve already learned how to access data directly from the NASA Earthdata Cloud?\n\nConsistent access patterns to EOSDIS holdings make cross-data center data access easier\nData reduction services allow us to request only the data we want, in the format and projection we want\nAnalysis Ready Data and cloud access will help reduce time-to-science\nCommunity Development helps reduce the barriers for re-use of code and sharing of domain knowledge\n\nSee more on the Earthdata Harmony landing page, including documentation on the Harmony API itself.\n\n\nObjectives\n\nConceptualize the data transformation service types and offerings provided by NASA Earthdata, including Harmony.\nPractice skills learned from the introductory CMR tutorial to discover what access and service options exist for a given data set, as well as variable metadata.\nUtilize the Harmony-py library to request subsetted MODIS L2 Sea Surface Temperature data over the Gulf of Mexico.\nRead Harmony subsetted outputs directly into xarray." + }, + { + "objectID": "examples/Transform-workflow.html#discover-service-options-for-a-given-data-set", + "href": "examples/Transform-workflow.html#discover-service-options-for-a-given-data-set", + "title": "Earthdata Cloud Exploration: Direct Access and Harmony workflow", + "section": "Discover service options for a given data set", + "text": "Discover service options for a given data set\n\nFirst, what do we mean by a “service”?\nIn the context of NASA Earthdata, we are usually referring to a service as any data transformation or customization process that packages or delivers data in a way that makes it easier to work with compared to how the data are natively archived at NASA EOSDIS. Basic customization options may include: * Subsetting (cropping) the data by: * Variable * Spatial boundary, * Temporal range * Reformatting * For example: From NetCDF-4 to Cloud Optimized GeoTIFF * Reprojection and/or Resampling * For example: From Sinusoidal to Polar Stereographic * Mosaicking * Aggregating\nA few main types or pathways for services that are commonly supported across the NASA DAACs include: * NASA Global Imagery Browse Service * Web services providing imagery, much of which is updated daily, to broaden accessibility of NASA EOSDIS data to the media and public. * OPeNDAP * The Open-source Project for a Network Data Access Protocol is a NASA community standard DAP that provides a simple way to access and work with data over the internet. OPeNDAP’s client/server software allows us to subset and reformat data using an internet browser, command line interface, and other applications. * Harmony * In the most basic sense, Harmony is an Application Programming Interface, or API, allowing us to request customization options described above, which are then processed and returned as file outputs. Harmony helps to reduce pre-processing steps so we can spend less time preparing the data, and more time doing science.\nNote: These service offerings are unique to each NASA EOSDIS dataset.\nWhy is this?\nDue to varying levels of service, cloud migration status, and unique characteristics of the datasets themselves, not all service options are provided for all datasets. Therefore it is important to first explore a given dataset’s metadata to discover what service options are provided.\n\n\nNote that the full Harmony tutorial from the 2021 Earthdata Cloud Hackathon demonstrates service and variable discovery, but this was removed here for simplification." + }, + { + "objectID": "examples/Transform-workflow.html#using-harmony-py-to-subset-data", + "href": "examples/Transform-workflow.html#using-harmony-py-to-subset-data", + "title": "Earthdata Cloud Exploration: Direct Access and Harmony workflow", + "section": "Using Harmony-Py to subset data", + "text": "Using Harmony-Py to subset data\nHarmony-Py provides a pip installable Python alternative to directly using Harmony’s RESTful API to make it easier to request data and service options, especially when interacting within a Python Jupyter Notebook environment.\nThe next steps are adopted from the introduction tutorial notebook provided in the Harmony-Py library:\n\nCreate Harmony Client object\nFirst, we need to create a Harmony Client, which is what we will interact with to submit and inspect a data request to Harmony, as well as to retrieve results.\nWhen creating the Client, we need to provide Earthdata Login credentials, which are required to access data from NASA EOSDIS. This basic line below assumes that we have a .netrc available.\n\nharmony_client = Client()\n\n\n\nCreate Harmony Request\nThe following are common request parameters:\n\ncollection: Required parameter. This is the NASA EOSDIS collection, or data product. There are two options for inputting a collection of interest:\n\nProvide a concept ID (e.g. C1940473819-POCLOUD)\nData product short name (e.g. MODIS_A-JPL-L2P-v2019.0).\n\nspatial: Bounding box spatial constraints on the data. The Harmony Bbox class accepts spatial coordinates as decimal degrees in w, s, e, n order, where longitude = -180, 180 and latitude = -90, 90.\ntemporal: Date/time constraints on the data. The example below demonstrates temporal start and end ranges using the python datetime library.\n\nAs we identified above, only subsetting options are available for this dataset. If other service options such as reformatting are available for a given dataset, these can also be specified using Harmony-py: See the documentation for details on how to construct these parameters.\n\nrequest = Request(\n collection=Collection(id=short_name),\n spatial=BBox(-97.77667,21.20806,-83.05197,30.16605),\n temporal={\n 'start': dt.datetime(2021, 8, 20),\n 'stop': dt.datetime(2021, 8, 21),\n },\n)\n\n\n\nSubmit request\nNow that the request is created, we can now submit it to Harmony using the Harmony Client object. A job id is returned, which is a unique identifier that represents the submitted request.\n\njob_id = harmony_client.submit(request)\njob_id\n\n'42f70e4f-5e6a-4473-8a5a-4ce56155ce92'\n\n\n\n\nCheck request status\nDepending on the size of the request, it may be helpful to wait until the request has completed processing before the remainder of the code is executed. The wait_for_processing() method will block subsequent lines of code while optionally showing a progress bar.\n\nharmony_client.wait_for_processing(job_id, show_progress=True)\n\n [ Processing: 100% ] |###################################################| [|]\n\n\n\n\nView Harmony job response and output URLs\nOnce the data request has finished processing, we can view details on the job that was submitted to Harmony, including the API call to Harmony, and informational messages on the request if available.\nresult_json() calls wait_for_processing() and returns the complete job in JSON format once processing is complete.\n\ndata = harmony_client.result_json(job_id)\npprint(data)\n\n{'createdAt': '2022-03-10T01:08:45.232Z',\n 'jobID': '42f70e4f-5e6a-4473-8a5a-4ce56155ce92',\n 'links': [{'href': 'https://harmony.earthdata.nasa.gov/stac/42f70e4f-5e6a-4473-8a5a-4ce56155ce92/',\n 'rel': 'stac-catalog-json',\n 'title': 'STAC catalog',\n 'type': 'application/json'},\n {'bbox': [-97.8, 21.2, -90.8, 22.5],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820172501-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T17:29:59.000Z',\n 'start': '2021-08-20T17:25:01.000Z'},\n 'title': '20210820172501-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-97.8, 21.2, -86.8, 30.2],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820172001-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T17:24:59.000Z',\n 'start': '2021-08-20T17:20:01.000Z'},\n 'title': '20210820172001-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-88.2, 21.2, -83.1, 28.3],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820154501-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T15:49:59.000Z',\n 'start': '2021-08-20T15:45:01.000Z'},\n 'title': '20210820154501-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-87.1, 28, -83.1, 30.2],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820154001-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T15:44:59.000Z',\n 'start': '2021-08-20T15:40:01.000Z'},\n 'title': '20210820154001-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-97.8, 21.2, -94, 30.2],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820051501-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T05:19:59.000Z',\n 'start': '2021-08-20T05:15:01.000Z'},\n 'title': '20210820051501-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'bbox': [-95.4, 21.2, -83.1, 30.2],\n 'href': 'https://harmony.earthdata.nasa.gov/service-results/harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820033500-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0_subsetted.nc4',\n 'rel': 'data',\n 'temporal': {'end': '2021-08-20T03:39:59.000Z',\n 'start': '2021-08-20T03:35:00.000Z'},\n 'title': '20210820033500-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0_subsetted.nc4',\n 'type': 'application/x-netcdf4'},\n {'href': 'https://harmony.earthdata.nasa.gov/jobs/42f70e4f-5e6a-4473-8a5a-4ce56155ce92?linktype=https&page=1&limit=2000',\n 'rel': 'self',\n 'title': 'The current page',\n 'type': 'application/json'}],\n 'message': 'The job has completed successfully',\n 'numInputGranules': 6,\n 'progress': 100,\n 'request': 'https://harmony.earthdata.nasa.gov/MODIS_T-JPL-L2P-v2019.0/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(21.20806%3A30.16605)&subset=lon(-97.77667%3A-83.05197)&subset=time(%222021-08-20T00%3A00%3A00%22%3A%222021-08-21T00%3A00%3A00%22)',\n 'status': 'successful',\n 'updatedAt': '2022-03-10T01:09:26.492Z',\n 'username': 'amy.steiker'}\n\n\n\n\nDirect cloud access\nNote that the remainder of this tutorial will only succeed when running this notebook within the AWS us-west-2 region.\nHarmony data outputs can be accessed within the cloud using the s3 URLs and AWS credentials provided in the Harmony job response.\n\nRetrieve list of output URLs.\nThe result_urls() method calls wait_for_processing() and returns a list of the processed data URLs once processing is complete. You may optionally show the progress bar as shown below.\n\nresults = harmony_client.result_urls(job_id, link_type=LinkType.s3)\nurls = list(results)\npprint(urls)\n\n['s3://harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820172501-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820172001-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820154501-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820154001-JPL-L2P_GHRSST-SSTskin-MODIS_T-D-v02.0-fv01.0_subsetted.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820051501-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0_subsetted.nc4',\n 's3://harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820033500-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0_subsetted.nc4']\n\n\nWe’ll select a URL in the list to bring into xarray below, which matches the granule we opened directly from PO.DAAC’s archive bucket above. In Earthdata Search, we can see that this granule covers our area well.\n\nfilename = '20210820033500-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0_subsetted.nc4'\n\nurl = [url for url in urls if filename in url][0]\nurl\n\n's3://harmony-prod-staging/public/podaac/l2-subsetter/4419d8b3-2127-4c8a-a7ef-2071d589de19/20210820033500-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0_subsetted.nc4'\n\n\n\n\nAWS credential retrieval\nUsing aws_credentials you can retrieve the credentials needed to access the Harmony s3 staging bucket and its contents.\n\ncreds = harmony_client.aws_credentials()\n\n\n\n\nOpen staged files with s3fs and xarray\nWe use the AWS s3fs package to create a file system that can then be read by xarray:\n\ns3_fs = s3fs.S3FileSystem(\n key=creds['aws_access_key_id'],\n secret=creds['aws_secret_access_key'],\n token=creds['aws_session_token'],\n client_kwargs={'region_name':'us-west-2'},\n)\n\nNow that we have our s3 file system set, including our declared credentials, we’ll use that to open the url, and read in the file through xarray. This extra step is needed because xarray cannot open the S3 location directly. Instead, the S3 file object is passed to xarray, in order to then open the dataset.\n\nf = s3_fs.open(url, mode='rb')\nds = xr.open_dataset(f)\nds\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: (nj: 1228, ni: 660, time: 1)\nCoordinates:\n lat (nj, ni, time) float32 ...\n lon (nj, ni, time) float32 ...\n * time (time) datetime64[ns] 2021-08-20T03:35:00\nDimensions without coordinates: nj, ni\nData variables:\n sea_surface_temperature (time, nj, ni) float32 ...\n sst_dtime (time, nj, ni) timedelta64[ns] ...\n quality_level (time, nj, ni) float32 ...\n sses_bias (time, nj, ni) float32 ...\n sses_standard_deviation (time, nj, ni) float32 ...\n l2p_flags (time, nj, ni) int16 ...\n sea_surface_temperature_4um (time, nj, ni) float32 ...\n quality_level_4um (time, nj, ni) float32 ...\n sses_bias_4um (time, nj, ni) float32 ...\n sses_standard_deviation_4um (time, nj, ni) float32 ...\n wind_speed (time, nj, ni) float32 ...\n dt_analysis (time, nj, ni) float32 ...\nAttributes: (12/50)\n Conventions: CF-1.7, ACDD-1.3\n title: MODIS Terra L2P SST\n summary: Sea surface temperature retrievals produced a...\n references: GHRSST Data Processing Specification v2r5\n institution: NASA/JPL/OBPG/RSMAS\n history: MODIS L2P created at JPL PO.DAAC\\n2022-03-10 ...\n ... ...\n processing_level: L2P\n cdm_data_type: swath\n startDirection: Ascending\n endDirection: Ascending\n day_night_flag: Night\n history_json: [{\"date_time\": \"2022-03-10T01:09:25.100329+00...xarray.DatasetDimensions:nj: 1228ni: 660time: 1Coordinates: (3)lat(nj, ni, time)float32...long_name :latitudestandard_name :latitudeunits :degrees_northvalid_min :-90.0valid_max :90.0comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[810480 values with dtype=float32]lon(nj, ni, time)float32...long_name :longitudestandard_name :longitudeunits :degrees_eastvalid_min :-180.0valid_max :180.0comment :geographical coordinates, WGS84 projectioncoverage_content_type :coordinate[810480 values with dtype=float32]time(time)datetime64[ns]2021-08-20T03:35:00long_name :reference time of sst filestandard_name :timecomment :time of first sensor observationcoverage_content_type :coordinatearray(['2021-08-20T03:35:00.000000000'], dtype='datetime64[ns]')Data variables: (12)sea_surface_temperature(time, nj, ni)float32...long_name :sea surface temperaturestandard_name :sea_surface_skin_temperatureunits :kelvinvalid_min :-1000valid_max :10000comment :sea surface temperature from thermal IR (11 um) channelssource :NASA and University of Miamicoverage_content_type :physicalMeasurement[810480 values with dtype=float32]sst_dtime(time, nj, ni)timedelta64[ns]...long_name :time difference from reference timevalid_min :-32767valid_max :32767comment :time plus sst_dtime gives seconds after 00:00:00 UTC January 1, 1981coverage_content_type :referenceInformation[810480 values with dtype=timedelta64[ns]]quality_level(time, nj, ni)float32...long_name :quality level of SST pixelvalid_min :0valid_max :5comment :thermal IR SST proximity confidence value; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valueflag_values :[0 1 2 3 4 5]flag_meanings :no_data bad_data worst_quality low_quality acceptable_quality best_qualitycoverage_content_type :qualityInformation[810480 values with dtype=float32]sses_bias(time, nj, ni)float32...long_name :SSES bias error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :thermal IR SST bias error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[810480 values with dtype=float32]sses_standard_deviation(time, nj, ni)float32...long_name :SSES standard deviation error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :thermal IR SST standard deviation error; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[810480 values with dtype=float32]l2p_flags(time, nj, ni)int16...long_name :L2P flagsvalid_min :0valid_max :16comment :These flags can be used to further filter data variablesflag_meanings :microwave land ice lake riverflag_masks :[ 1 2 4 8 16]coverage_content_type :qualityInformation[810480 values with dtype=int16]sea_surface_temperature_4um(time, nj, ni)float32...long_name :sea surface temperatureunits :kelvinvalid_min :-1000valid_max :10000comment :sea surface temperature from mid-IR (4 um) channels; non L2P core fieldcoverage_content_type :physicalMeasurement[810480 values with dtype=float32]quality_level_4um(time, nj, ni)float32...long_name :quality level of SST pixelvalid_min :0valid_max :5comment :mid-IR SST proximity confidence value; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valueflag_values :[0 1 2 3 4 5]flag_meanings :no_data bad_data worst_quality low_quality acceptable_quality best_qualitycoverage_content_type :qualityInformation[810480 values with dtype=float32]sses_bias_4um(time, nj, ni)float32...long_name :SSES bias error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :mid-IR SST bias error; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[810480 values with dtype=float32]sses_standard_deviation_4um(time, nj, ni)float32...long_name :SSES standard deviation error based on proximity confidence flagsunits :kelvinvalid_min :-127valid_max :127comment :mid-IR SST standard deviation error; non L2P core field; signed byte array: WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported valuecoverage_content_type :auxiliaryInformation[810480 values with dtype=float32]wind_speed(time, nj, ni)float32...long_name :10m wind speedstandard_name :wind_speedunits :m s-1valid_min :-127valid_max :127comment :Wind at 10 meters above the sea surfacesource :TBD. Placeholder. Currently emptygrid_mapping :TBDtime_offset :2.0height :10 mcoverage_content_type :auxiliaryInformation[810480 values with dtype=float32]dt_analysis(time, nj, ni)float32...long_name :deviation from SST reference climatologyunits :kelvinvalid_min :-127valid_max :127comment :TBDsource :TBD. Placeholder. Currently emptycoverage_content_type :auxiliaryInformation[810480 values with dtype=float32]Attributes: (50)Conventions :CF-1.7, ACDD-1.3title :MODIS Terra L2P SSTsummary :Sea surface temperature retrievals produced at the NASA OBPG for the MODIS Aqua sensor. These have been reformatted to GHRSST GDS specifications by the JPL PO.DAACreferences :GHRSST Data Processing Specification v2r5institution :NASA/JPL/OBPG/RSMAShistory :MODIS L2P created at JPL PO.DAAC\n2022-03-10 01:09:25.100285 l2ss-py v1.3.1 (bbox=[[-97.77667, -83.05197], [21.20806, 30.16605]] cut=True)comment :L2P Core without DT analysis or other ancillary fields; Night, Start Node:Ascending, End Node:Ascending; WARNING Some applications are unable to properly handle signed byte values. If values are encountered > 127, please subtract 256 from this reported value; Quicklooklicense :GHRSST and PO.DAAC protocol allow data use as free and open.id :MODIS_T-JPL-L2P-v2019.0naming_authority :org.ghrsstproduct_version :2019.0uuid :f6e1f61d-c4a4-4c17-8354-0c15e12d688bgds_version_id :2.0netcdf_version_id :4.1date_created :20210820T153042Zfile_quality_level :3spatial_resolution :1kmstart_time :20210820T033500Ztime_coverage_start :20210820T033500Zstop_time :20210820T033959Ztime_coverage_end :20210820T033959Znorthernmost_latitude :34.0296southernmost_latitude :12.9895easternmost_longitude :-68.4372westernmost_longitude :-95.5518source :MODIS sea surface temperature observations for the OBPGplatform :Terrasensor :MODISmetadata_link :http://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=MODIS_T-JPL-L2P-v2019.0keywords :Oceans > Ocean Temperature > Sea Surface Temperaturekeywords_vocabulary :NASA Global Change Master Directory (GCMD) Science Keywordsstandard_name_vocabulary :NetCDF Climate and Forecast (CF) Metadata Conventiongeospatial_lat_units :degrees_northgeospatial_lat_resolution :0.01geospatial_lon_units :degrees_eastgeospatial_lon_resolution :0.01acknowledgment :The MODIS L2P sea surface temperature data are sponsored by NASAcreator_name :Ed Armstrong, JPL PO.DAACcreator_email :edward.m.armstrong@jpl.nasa.govcreator_url :http://podaac.jpl.nasa.govproject :Group for High Resolution Sea Surface Temperaturepublisher_name :The GHRSST Project Officepublisher_url :http://www.ghrsst.orgpublisher_email :ghrsst-po@nceo.ac.ukprocessing_level :L2Pcdm_data_type :swathstartDirection :AscendingendDirection :Ascendingday_night_flag :Nighthistory_json :[{\"date_time\": \"2022-03-10T01:09:25.100329+00:00\", \"derived_from\": \"https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MODIS_T-JPL-L2P-v2019.0/20210820033500-JPL-L2P_GHRSST-SSTskin-MODIS_T-N-v02.0-fv01.0.nc\", \"program\": \"l2ss-py\", \"version\": \"1.3.1\", \"parameters\": \"bbox=[[-97.77667, -83.05197], [21.20806, 30.16605]] cut=True\", \"program_ref\": \"https://cmr.earthdata.nasa.gov:443/search/concepts/S1962070864-POCLOUD\", \"$schema\": \"https://harmony.earthdata.nasa.gov/schemas/history/0.1.0/history-v0.1.0.json\"}]\n\n\n\n\nPlot the data\nUse the xarray built in plotting function to create a simple plot along the x and y dimensions of the dataset:\n\nds.sea_surface_temperature.plot() ;\n\n\n\n\n\n\nSummary\nThis notebook demonstrated an abbreviated and simplified workflow to explore access and subsetting options available through the Earthdata Cloud. There are several other options that can be used to work “in place” in the Earthdata Cloud, from data discovery to analysis-ready data, including:\n\nZarr-EOSDIS-Store\n\nThe zarr-eosdis-store library allows NASA EOSDIS Collections to be accessed efficiently by the Zarr Python library, provided they have a sidecar DMR++ metadata file generated.\nTutorial highlighting this library’s usage: https://nasa-openscapes.github.io/2021-Cloud-Hackathon/tutorials/09_Zarr_Access.html\n\nOPeNDAP\n\nHyrax provides direct access to subsetting of NASA data using Python or your favorite analysis tool\nTutorial highlighting OPeNDAP usage: https://nasa-openscapes.github.io/2021-Cloud-Hackathon/tutorials/06_S6_OPeNDAP_Access_Gridding.html\n\nEarthdata Library\n\nA Python library to search and access NASA Earthdata datasets, aiming to provide a simple way to access or download data without having to worry if a given dataset is on-prem or in the cloud." + }, + { + "objectID": "examples/LPDAAC/LPDAAC_ECOSTRESS_LSTE__Kerchunk.html", + "href": "examples/LPDAAC/LPDAAC_ECOSTRESS_LSTE__Kerchunk.html", + "title": "LP DAAC ECOSTRESS", + "section": "", + "text": "!mamba install kerchunk -y --quiet\n\n\n!mamba install h5py=3.2 -y --quiet # Default version in this environment does not work. Must update\n\n\nimport numpy as np\nimport xarray as xr\nimport matplotlib.pyplot as plt\nimport s3fs\nimport datetime as dt\nimport logging\nimport fsspec\nimport ujson\nimport requests\nfrom tqdm import tqdm\nfrom glob import glob\nimport os\nimport hvplot.xarray\n\n# The xarray produced from the reference file throws a SerializationWarning for each variable. Will need to explore why\n#import warnings\n#warnings.simplefilter(\"ignore\")\n\n\n\n\n\n\n\n\n\n\n\n\nfrom kerchunk.hdf import SingleHdf5ToZarr\nfrom kerchunk.combine import MultiZarrToZarr\n\n\ns3_cred_endpoint = {\n 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n 'ornldaac':'https://data.ornldaac.earthdata.nasa.gov/s3credentials',\n 'gesdisc':'https://data.gesdisc.earthdata.nasa.gov/s3credentials'\n}\n\n\ndef get_temp_creds():\n temp_creds_url = s3_cred_endpoint['lpdaac']\n return requests.get(temp_creds_url).json()\n\n\ntemp_creds_req = get_temp_creds()\n\nDirect Access a Single file\n\nfs = s3fs.S3FileSystem(\n anon=False,\n key=temp_creds_req['accessKeyId'],\n secret=temp_creds_req['secretAccessKey'],\n token=temp_creds_req['sessionToken']\n)\n\n\nurl = 's3://lp-prod-protected/ECO_L1B_GEO.002/ECOv002_L1B_GEO_21547_021_20220424T215449_0700_01/ECOv002_L1B_GEO_21547_021_20220424T215449_0700_01.h5'\n\n\ns3_file_obj = fs.open(url, mode='rb')\n\n\nimport h5py\n\n\nh5_file = h5py.File(s3_file_obj)\n\n\n#list(h5_file['Geolocation']['latitude'])\n\n\n%%time\nxr_ds = xr.open_dataset(s3_file_obj, chunks='auto', engine='h5netcdf', backend_kwargs={\"mask_and_scale\" : False, \"decode_times\" : False, \"decode_timedelta\" : False, \"use_cftime\" : False, \"decode_coords\" : False})\nxr_ds\n\nCPU times: user 134 ms, sys: 35 ms, total: 169 ms\nWall time: 534 ms\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<xarray.Dataset>\nDimensions: ()\nData variables:\n *empty*xarray.DatasetDimensions:Coordinates: (0)Data variables: (0)Attributes: (0)\n\n\n\nSpecify a list of S3 URLs (MERRA2, 05/01/2019 - 05/31/2019)\n\nurls = ['s3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190501.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190502.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190503.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190504.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190505.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190506.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190507.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190508.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190509.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190510.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190511.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190512.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190513.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190514.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190515.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190516.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190517.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190518.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190519.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190520.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190521.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190522.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190523.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190524.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190525.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190526.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190527.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190528.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190529.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190530.nc4',\n's3://gesdisc-cumulus-prod-protected/MERRA2/M2T1NXSLV.5.12.4/2019/05/MERRA2_400.tavg1_2d_slv_Nx.20190531.nc4']\n\nGenerate the reference kerchunk files. These files can take a little time to generate. Do not run if JSON files already exist in ./jsons/ directory.\n\nso = dict(\n mode=\"rb\",\n anon=False,\n default_fill_cache=False,\n default_cache_type=\"none\"\n)\n\n#references = []\nfor u in urls:\n with fs.open(u, **so) as infile:\n h5chunks = SingleHdf5ToZarr(infile, u, inline_threshold=300)\n with open(f\"jsons/{u.split('/')[-1]}.json\", 'wb') as outfile:\n translate = h5chunks.translate()\n outfile.write(ujson.dumps(translate).encode())\n #references.append(translate)\n\nCreate a list with the paths to the reference files\n\nfs_ref_list = fsspec.filesystem('file')\n\n\nreference_list = sorted([x for x in fs_ref_list.ls('jsons') if '.json' in x])\nreference_list\n\nOpen the first reference file to read into an xarray dataset\n\nwith open(reference_list[0]) as j:\n reference = ujson.load(j)\n\nSet configurations options\n\ns_opts = {'skip_instance_cache':True} #json\nr_opts = {'anon':False, \n 'key':temp_creds_req['accessKeyId'], \n 'secret':temp_creds_req['secretAccessKey'], \n 'token':temp_creds_req['sessionToken']} #ncfiles\n\n\nfs_single = fsspec.filesystem(\"reference\", \n fo=reference, \n ref_storage_args=s_opts,\n remote_protocol='s3', \n remote_options=r_opts)\n\nRead in a single reference object. Get lots of SerializationWarnings. Ignored using the warning package, but the fill value, data range, min value, and max value DO NOT match the source file.\n\n%%time\nm = fs_single.get_mapper(\"\")\nds_single = xr.open_dataset(m, engine=\"zarr\", backend_kwargs={'consolidated':False}, chunks={})\nds_single\n\n\n#ds_single\n\n\nCombine the single reference files created above into a single time series reference file THIS DID NOT WORK\n\n# mzz = MultiZarrToZarr(\n# reference_list,\n# remote_protocol=\"s3\",\n# remote_options=r_opts,\n# concat_dims=[\"time\"]\n# )\n\n# out = mzz.translate()\n\n\n# fs_mzz = fsspec.filesystem(\"reference\",\n# fo=out,\n# ref_storage_args=s_opts,\n# remote_protocol='s3',\n# remote_options=r_opts)\n\n\n# m = fs_mzz.get_mapper(\"\")\n# ds_multi = xr.open_dataset(m, engine=\"zarr\", backend_kwargs={'consolidated':False}, chunks={})\n\n\nCombine the single reference files created above into a single time series reference file THIS DID WORK\n\n%%time\nds_k =[]\n\nfor ref in reference_list:\n s_opts = s_opts\n r_opts = r_opts\n fs = fsspec.filesystem(\n \"reference\", \n fo=ref, \n ref_storage_args=s_opts,\n remote_protocol='s3', \n remote_options=r_opts)\n m = fs.get_mapper(\"\")\n ds_k.append(xr.open_dataset(m, engine=\"zarr\", backend_kwargs={'consolidated':False}, chunks={}))\n \nds_multi = xr.concat(ds_k, dim='time')\n \nds_multi\n\nAgains, the fill value, data range, min value, and max value DO NOT match the source file\n\n#ds_multi = xr.concat(ds_k, dim='time')\n\n\n#ds_multi\n\n\nds_multi['T500']\n\n\n# Commenting for quarto site render\n# ds_multi['T500'].hvplot.image(x='lon', y='lat')" } ] \ No newline at end of file diff --git a/sitemap.xml b/sitemap.xml index 5ddaa0b9..53bbc5fd 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -1,327 +1,327 @@ - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/LICENSE.html - 2024-04-30T23:50:09.565Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html + 2024-05-28T19:56:17.737Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/workshops/setup.html - 2024-04-30T23:50:08.737Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/LPDAAC/Find_and_Access_HLS_PointBuffer.html + 2024-05-28T19:56:05.385Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/data_discovery_cmr.html - 2024-04-30T23:50:07.509Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/USGS_Landsat/Landsat_C2_Search_Access.html + 2024-05-28T19:56:02.525Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/harmony_subsetting.html - 2024-04-30T23:50:06.377Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html + 2024-05-28T19:56:00.553Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/sentinel-6_opendap_access_gridding.html - 2024-04-30T23:50:04.205Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/ORNL/Data_Access__Direct_S3_Access__ORNL_DAYMET.html + 2024-05-28T19:55:59.749Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/data_access_direct_S3.html - 2024-04-30T23:50:02.881Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html + 2024-05-28T19:55:57.113Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/nasa_earthdata_authentication.html - 2024-04-30T23:49:59.545Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/workshops/setup.html + 2024-05-28T19:55:53.753Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/Above_Ground_Biomass_Density.html - 2024-04-30T23:49:58.609Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/data_discovery_cmr-stac_api.html + 2024-05-28T19:55:52.433Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/index.html - 2024-04-30T23:49:57.497Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/data_discovery_cmr.html + 2024-05-28T19:55:50.637Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/Earthdata-cloud-clinic.html - 2024-04-30T23:49:56.369Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/nasa_earthdata_authentication.html + 2024-05-28T19:55:49.213Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html - 2024-04-30T23:49:54.269Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/xarray.html + 2024-05-28T19:55:47.837Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html - 2024-04-30T23:49:53.161Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/data_access_direct_S3.html + 2024-05-28T19:55:46.813Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/Data_stories_template.html - 2024-04-30T23:49:48.681Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/use_apis.html + 2024-05-28T19:55:43.437Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/index.html - 2024-04-30T23:49:47.273Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/find-data/find-python.html + 2024-05-28T19:55:42.593Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/in-development/CMR-STAC-Search.html - 2024-04-30T23:49:46.554Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/find-data/find-r.html + 2024-05-28T19:55:41.725Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/in-development/earthdata-python-r-handoff.html - 2024-04-30T23:49:45.090Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/reproject_regrid.html + 2024-05-28T19:55:39.869Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/workflow.html - 2024-04-30T23:49:44.146Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/access-data/access-local-opendap.html + 2024-05-28T19:55:38.745Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/quarto-website-tutorial.html - 2024-04-30T23:49:42.578Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/how_to_guide_parking_lot.html + 2024-05-28T19:55:37.621Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/calendar-project-management.html - 2024-04-30T23:49:41.842Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/plot.html + 2024-05-28T19:55:36.517Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/troubleshooting.html - 2024-04-30T23:49:41.078Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/reformat.html + 2024-05-28T19:55:28.817Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/how_to_guide_parking_lot.html - 2024-04-30T23:49:40.378Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/aggregate.html + 2024-05-28T19:55:27.673Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/Earthdata_Cloud__Open-Science-Tutorial.html - 2024-04-30T23:49:39.478Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/in-development/earthdata-python-r-handoff.html + 2024-05-28T19:55:26.465Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/access-data/access-local-opendap.html - 2024-04-30T23:49:31.998Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/in-development/matlab.html + 2024-05-28T19:55:25.529Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/access-data/access-cloud-python.html - 2024-04-30T23:49:30.906Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/environment-setup/index.html + 2024-05-28T19:55:24.405Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/use_apis.html - 2024-04-30T23:49:29.814Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/LICENSE.html + 2024-05-28T19:55:23.617Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/find-data/find-python.html - 2024-04-30T23:49:28.078Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/workflow.html + 2024-05-28T19:55:22.865Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/find-data/earthdata_search.html - 2024-04-30T23:49:27.230Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/environment.html + 2024-05-28T19:55:21.297Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/aggregate.html - 2024-04-30T23:49:25.962Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/calendar-project-management.html + 2024-05-28T19:55:20.501Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/reproject_regrid.html - 2024-04-30T23:49:24.818Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/onboarding.html + 2024-05-28T19:55:19.789Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/when-to-cloud.html - 2024-04-30T23:49:24.122Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/leading-workshops/add-folks-to-2i2c-github-teams.html + 2024-05-28T19:55:18.977Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/leading-workshops/index.html - 2024-04-30T23:49:23.270Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/index.html + 2024-05-28T19:55:17.909Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html - 2024-04-30T23:49:22.482Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/appendix/appendix-alternate-in-cloud-access/Access_COG.html + 2024-05-28T19:55:17.021Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html - 2024-04-30T23:49:21.398Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/appendix/appendix-alternate-in-cloud-access/Multi-File_Direct_S3_Access_NetCDF_Example.html + 2024-05-28T19:55:15.669Z https://nasa-openscapes.github.io/earthdata-cloud-cookbook/appendix/authentication.html - 2024-04-30T23:49:20.326Z + 2024-05-28T19:55:14.801Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/glossary.html - 2024-04-30T23:49:19.362Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/policies-usage/index.html + 2024-05-28T19:55:13.765Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/PODAAC/PODAAC_ECCO_SSH__Kerchunk.html - 2024-04-30T23:49:18.574Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/Hurricanes_Wind_and_Sea_Surface_Temperature.html + 2024-05-28T19:55:12.801Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/ORNL/Data_Access__Direct_S3_Access__ORNL_DAYMET.html - 2024-04-30T23:49:15.362Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/Data_stories_template.html + 2024-05-28T19:55:11.725Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/NSIDC/ICESat2-CMR-OnPrem-vs-Cloud.html - 2024-04-30T23:49:11.538Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/index.html + 2024-05-28T19:55:10.865Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/GESDISC/GESDISC_MERRA2_tavg1_2d_flx_Nx__Kerchunk.html - 2024-04-30T23:49:10.402Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/Earthdata-cloud-clinic.html + 2024-05-28T19:55:06.421Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/LPDAAC/LPDAAC_ECOSTRESS_LSTE__Kerchunk.html - 2024-04-30T23:48:58.398Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/Observing_Seasonal_Ag_Changes.html + 2024-05-28T19:55:04.557Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/our-cookbook.html - 2024-04-30T23:48:55.402Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/matlab.html + 2024-05-28T19:55:01.709Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/USGS_Landsat/Landsat_C2_Search_Access.html - 2024-04-30T23:48:57.278Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/04_On-Prem_Cloud.html + 2024-05-28T19:55:03.105Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/LPDAAC/Find_and_Access_HLS_PointBuffer.html - 2024-04-30T23:48:59.538Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/fair-workflow-geoweaver-demo.html + 2024-05-28T19:55:05.685Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/NSIDC/ICESat2-CMR-AWS-S3.html - 2024-04-30T23:49:11.058Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/Sea_Level_Rise.html + 2024-05-28T19:55:10.469Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/Transform-workflow.html - 2024-04-30T23:49:13.174Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/Earthdata_Cloud__Data_Access_OPeNDAP_Example.html + 2024-05-28T19:55:11.389Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/ORNL/ORNL_DAYMET__Kerchunk.html - 2024-04-30T23:49:15.714Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/earthaccess-demo.html + 2024-05-28T19:55:12.285Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/index.html - 2024-04-30T23:49:18.882Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/Above_Ground_Biomass_Density.html + 2024-05-28T19:55:13.405Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/appendix/index.html - 2024-04-30T23:49:19.794Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/glossary.html + 2024-05-28T19:55:14.253Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/appendix/appendix-alternate-in-cloud-access/Access_COG.html - 2024-04-30T23:49:20.974Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/appendix/index.html + 2024-05-28T19:55:15.233Z https://nasa-openscapes.github.io/earthdata-cloud-cookbook/appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.html - 2024-04-30T23:49:22.046Z - - - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/leading-workshops/add-folks-to-2i2c-github-teams.html - 2024-04-30T23:49:22.926Z + 2024-05-28T19:55:16.309Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/environment-setup/index.html - 2024-04-30T23:49:23.634Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/appendix/appendix-alternate-in-cloud-access/Earthdata_Cloud__Single_File__Direct_S3_Access_NetCDF4_Example.html + 2024-05-28T19:55:17.461Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/policies-usage/index.html - 2024-04-30T23:49:24.486Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/leading-workshops/index.html + 2024-05-28T19:55:18.257Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/using-s3-storage.html - 2024-04-30T23:49:25.634Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/index.html + 2024-05-28T19:55:19.329Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/read_data.html - 2024-04-30T23:49:26.734Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/community.html + 2024-05-28T19:55:20.109Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/find-data/programmatic.html - 2024-04-30T23:49:27.758Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/setup.html + 2024-05-28T19:55:20.941Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/find-data/find-r.html - 2024-04-30T23:49:29.438Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/quarto-website-tutorial.html + 2024-05-28T19:55:21.609Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/reformat.html - 2024-04-30T23:49:30.122Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/troubleshooting.html + 2024-05-28T19:55:23.229Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/access-data/access-local-python.html - 2024-04-30T23:49:31.682Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/our-cookbook.html + 2024-05-28T19:55:24.029Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/plot.html - 2024-04-30T23:49:32.318Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/when-to-cloud.html + 2024-05-28T19:55:24.885Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/subset.html - 2024-04-30T23:49:40.046Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/in-development/index.html + 2024-05-28T19:55:25.837Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/environment.html - 2024-04-30T23:49:40.726Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/in-development/CMR-STAC-Search.html + 2024-05-28T19:55:27.341Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/setup.html - 2024-04-30T23:49:41.482Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/using-s3-storage.html + 2024-05-28T19:55:28.501Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/onboarding.html - 2024-04-30T23:49:42.278Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/Earthdata_Cloud__Open-Science-Tutorial.html + 2024-05-28T19:55:36.149Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/index.html - 2024-04-30T23:49:42.918Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/read_data.html + 2024-05-28T19:55:37.285Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/contributing/community.html - 2024-04-30T23:49:44.458Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/access-data/access-local-python.html + 2024-05-28T19:55:38.413Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/in-development/matlab.html - 2024-04-30T23:49:45.702Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/access-data/access-cloud-python.html + 2024-05-28T19:55:39.529Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/in-development/index.html - 2024-04-30T23:49:46.858Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/find-data/earthdata_search.html + 2024-05-28T19:55:40.377Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/fair-workflow-geoweaver-demo.html - 2024-04-30T23:49:48.349Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/find-data/programmatic.html + 2024-05-28T19:55:42.261Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/Sea_Level_Rise.html - 2024-04-30T23:49:52.641Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/subset.html + 2024-05-28T19:55:43.065Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/matlab.html - 2024-04-30T23:49:53.769Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/on-prem_cloud.html + 2024-05-28T19:55:44.217Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/Observing_Seasonal_Ag_Changes.html - 2024-04-30T23:49:55.641Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/cof-zarr-reformat.html + 2024-05-28T19:55:47.325Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/04_On-Prem_Cloud.html - 2024-04-30T23:49:57.141Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/sentinel-6_opendap_access_gridding.html + 2024-05-28T19:55:48.633Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/earthaccess-demo.html - 2024-04-30T23:49:58.017Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/harmony_subsetting.html + 2024-05-28T19:55:50.005Z https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/zarr-eosdis-store.html - 2024-04-30T23:49:58.977Z + 2024-05-28T19:55:51.005Z https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/appeears_csv_cloud_access.html - 2024-04-30T23:50:00.377Z + 2024-05-28T19:55:53.281Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/cof-zarr-reformat.html - 2024-04-30T23:50:03.417Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/workshops/index.html + 2024-05-28T19:55:54.205Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/data_discovery_cmr-stac_api.html - 2024-04-30T23:50:05.581Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/ORNL/ORNL_DAYMET__Kerchunk.html + 2024-05-28T19:55:57.489Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/xarray.html - 2024-04-30T23:50:06.881Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/index.html + 2024-05-28T19:56:00.069Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/on-prem_cloud.html - 2024-04-30T23:50:08.289Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/NSIDC/ICESat2-CMR-AWS-S3.html + 2024-05-28T19:56:01.209Z - https://nasa-openscapes.github.io/earthdata-cloud-cookbook/workshops/index.html - 2024-04-30T23:50:09.181Z + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/Transform-workflow.html + 2024-05-28T19:56:04.225Z + + + https://nasa-openscapes.github.io/earthdata-cloud-cookbook/examples/LPDAAC/LPDAAC_ECOSTRESS_LSTE__Kerchunk.html + 2024-05-28T19:56:06.529Z diff --git a/when-to-cloud.html b/when-to-cloud.html index 548e2609..8019ae87 100644 --- a/when-to-cloud.html +++ b/when-to-cloud.html @@ -7,7 +7,7 @@ - + EarthData Cloud Cookbook - When To ‘Cloud’