Skip to content

Commit

Permalink
merged with development
Browse files Browse the repository at this point in the history
  • Loading branch information
atteggiani committed Aug 10, 2023
2 parents 28cde6c + cea3bfc commit a466110
Show file tree
Hide file tree
Showing 82 changed files with 1,769 additions and 2,380 deletions.
29 changes: 26 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,22 @@ https://access-hive.org.au/
If you wish to add documentation to the ACCESS-Hive website see the [contribution guide](https://access-hive.org.au/about/contribute/) for instructions.

# How to Contribute [Draft]
- Create a branch from the `development` branch of ACCESS-HIVE repo, for instance, `dev/jasmeen/legacy release`. Use the following two commands to create a new branch and push it to github respectively:

- Pull the latest version of `development` branch locally by using the following commands:

> Fetch the remote branches from github:
`git fetch`

> On terminal, switch to the `development` branch locally using the command:
`git switch development`

> Pull the latest changes from remote `development` branch locally:
`git pull`

- Once the latest version of `development` branch is pulled locally, use the following two commands to create the new branch and push it to github respectively:

`git checkout -b dev/jasmeen/legacy-release`

Expand All @@ -28,10 +43,18 @@ If you wish to add documentation to the ACCESS-Hive website see the [contributio
`git commit -m "First commit"`

`git push`

- While working locally on the branch, please make sure to regularly pull changes from remote `development` branch into your branch, using the command:

`git pull origin development`

This would make sure that the local branch `dev/jasmeen/legacy-release` is always in sync with the latest changes in the remote `development` branch.

- Once the changes on the local branch `dev/jasmeen/legacy-release` is ready to be integrated with the `development` branch, create a pull request on github by changing the `base` to `development`. (Please find the below screenshot). This would create a pr on the `development` branch.

- On github, create a pull request and making sure to change the `base` to `development`. (Please find the below screenshot). This would create a pr on the `development` branch.
![Screenshot 2023-06-02 at 2 05 13 pm](https://github.com/ACCESS-Hive/access-hive.github.io/assets/42607679/ec141fc9-ee00-4a84-ae5a-081761400765)

![Screenshot 2023-06-02 at 2 05 13 pm](https://github.com/ACCESS-Hive/access-hive.github.io/assets/42607679/ec141fc9-ee00-4a84-ae5a-081761400765)
- When creating a pull request (PR) please also assign a reviewer to avoid delays. For technical content please assign an expert reviewer.

# License
The ACCESS-Hive site is covered by [the CC-BY 4.0 license][License].
Expand Down
2 changes: 1 addition & 1 deletion docs/about/License.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# <div class="highlight-bg"> License </div>
# License

The ACCESS-Hive is made available under the [Creative Commons Attribution
license][cc-by-human]. The following is a human-readable summary of
Expand Down
2 changes: 1 addition & 1 deletion docs/about/code_of_conduct.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# <div class="highlight-bg"> Code of Conduct </div>
# Code of Conduct

ACCESS-Hive is an open community supported effort. For it to be successful it must be a welcoming and inclusive space so that everyone in the community feels able to contribute.

Expand Down
2 changes: 1 addition & 1 deletion docs/about/contact.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# <div class="highlight-bg"> Contact </div>
# Contact

<img src = "../../assets/contact-img.jpg" class="rectangular-img img-cover with-border"></img>

Expand Down
2 changes: 1 addition & 1 deletion docs/about/glossaries.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# <div class="highlight-bg"> Glossary and Useful Terms </div>
# Glossary and Useful Terms

Here you can find a compilation of common terms and acronyms used by the Australian Community Climate and Earth System Simulator (ACCESS) community:

Expand Down
11 changes: 11 additions & 0 deletions docs/about/how_to_contribute.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# How to Contribute

ACCESS-Hive is an open portal for the ACCESS user community, hosted by <a href="https://www.access-nri.org.au" target="_blank">ACCESS-NRI</a>.

Contributions to the ACCESS-Hive (ACCESS documentation) as well as the ACCESS-Hive Forum (Community exchange) are **encouraged by any member of the community**.
<br>
You can contribute in two ways:

1. Become part of the ACCESS user community by signing up and participate on the <a href="https://forum.access-hive.org.au" target="_blank">ACCESS-Hive Forum</a>.

2. Help us improve the ACCESS-Hive documentation by raising a <a href="https://github.com/ACCESS-Hive/access-hive.github.io/issues/new?assignees=&labels=External&projects=&template=simple-issue-template.md&title=" target="_blank">New issue</a> with your specific suggestion/content on the <a href="https://github.com/ACCESS-Hive/access-hive.github.io" target="_blank">ACCESS-Hive Github Repository</a>.
2 changes: 1 addition & 1 deletion docs/about/policies.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# <div class="highlight-bg">Policies</div>
# Policies

- [Procedures and Practices][pandp]: Contains documents outlining how ACCESS-NRI will function. These documents describe what users can expect and justifications for the decisions against criteria that are based on the values of the organisation.

Expand Down
12 changes: 6 additions & 6 deletions docs/about/user_support.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# <div class="highlight-bg"> User support </div>
# User support

## How to get help

Each entry on ACCESS-Hive links to another web site. There should be information on how to get help on the linked site. If there are no obvious channels for help, or the help is not adequate consider asking for assistance from fellow members of your community on the [ACCESS-Hive forum][access-hive-forum].
For resources developed by another party that ACCESS-Hive links to, there should be information on how to get help on the linked site. If there are no obvious channels for help, or the help is not adequate consider asking for assistance from fellow members of your community on the <a href="https://forum.access-hive.org.au" target="_blank">ACCESS-Hive Forum</a>.

In the case of ACCESS-NRI supported documentation and software, marked , if there is no information on how to get help, or your query is not appropriate for the support channels provided, please either ask on the [ACCESS-Hive forum][access-hive-forum] or [contact ACCESS-NRI directly][contact].
In the case of ACCESS-NRI supported documentation and software, please ask on the <a href="https://forum.access-hive.org.au" target="_blank">ACCESS-Hive Forum</a>.

[contact]: contact.md
[nci-web]: https://www.nci.org.au
[access-hive-forum]: https://forum.access-hive.org.au
How to get help and support, and what assistance is available on the forum is outlined in <a href="https://forum.access-hive.org.au/t/access-help-and-support/908" target="_blank">ACCESS Help and Support</a>.

There is also a <a href="https://forum.access-hive.org.au/t/support-faq-frequently-asked-questions/1021" target="_blank">Support FAQ</a> covering some of the more common questions you might have about what support is provided, how it is provided and what you can expect.
Binary file added docs/assets/getting_started/are_1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/getting_started/are_2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/model_evaluation/METplus_logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/model_evaluation/accessnri_intake.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/model_evaluation/intake_example.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/model_evaluation/netcdf_1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/model_evaluation/netcdf_example.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/model_evaluation/netcdf_example.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/models_flow_diagram.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/terminal_animation_switch_active.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
261 changes: 166 additions & 95 deletions docs/community_resources/community_med/community_data_processing.md
Original file line number Diff line number Diff line change
@@ -1,97 +1,168 @@
# Community Processing Data Processing Tools

<!-- {% include "call_contribute.md" %} -->


## Tools

### Kerchunk

[**Documentation**][kerchunk-doc] |
[**Sources**][kerchunk-source]

Kerchunk is a library that provides a unified way to represent a variety of chunked, compressed data formats (e.g. NetCDF/HDF5, GRIB2, TIFF, …), allowing efficient access to the data from traditional file systems or cloud object storage. It also provides a flexible way to create virtual datasets from multiple files.

### CMOR3

*Climate Model Output Rewriter Version 3*

[**Documentation**][cmor3-doc] |
[**Sources**][cmor3-source]

CMOR is used to produce CF-compliant netCDF files. The structure of the files created by CMOR and the metadata they contain fulfill the requirements of many of the climate community’s standard model experiments (which are referred to here as “MIPs” and include, for example, AMIP, PMIP, APE, and IPCC scenario runs).

### xMIP

[**Documentation**][xmip-doc] |
[**Tutorial on NCI**][xmip-tutorial] |
[**Sources**][xmip-source]

This package facilitates the cleaning, organization and interactive analysis of Model Intercomparison Projects (MIPs) within the Pangeo software stack.

### APP4 (The ACCESS Post Processor)

[**Documentation**][APP4-doc] |
[**Sources**][APP4-source]

The APP4 is a CMORisation tool designed to convert ACCESS model output to ESGF-compliant formats, primarily for publication to CMIP6. The code was originally built for CMIP5, and was further developed for CMIP6-era activities.
Uses CMOR3 and files created with the CMIP6 data request to generate CF-compliant files according to the CMIP6 data standards.

### ACCESS-Archiver

[**Documentation**][ACCESS-Archiver-doc] |
[**Sources**][ACCESS-Archiver-source]

The ACCESS Archiver is designed to archive model output from ACCESS simulations. It's focus is to copy ACCESS model output from its initial location to a secondary location (typically from `/scratch` to `/g/data`), while converting UM files to netCDF, compressing MOM/CICE files, and culling restart files to 10-yearly. Saves 50-80% of storage space due to conversion and compression.

### Synda

synda is a command line tool to search and download files from the Earth System Grid Federation (ESGF) archive.

### FluxnetLSM

[**Citation** [@Ukkola2017]][fluxnetlsm-cite] |
[**Sources**][fluxnetlsm-source]

R package for post-processing FLUXNET datasets for use in land surface modelling. Performs quality control and data conversion of FLUXNET data and collated site metadata. Supports FLUXNET2015, La Thuile, OzFlux and ICOS data releases.

### Metpy

https://unidata.github.io/MetPy/latest/examples/formats/index.html

[**Documentation**][metpy_docu] |
[**Sources**][metpy_docu-source]

MetPy is a collection of tools in Python for reading, visualizing, and performing calculations with weather data. MetPy supports Python >= 3.8 and is freely available under a permissive open source license.

Format types are: GINI Water Vapor Imagery, NEXRAD Level 3 File, and NEXRAD Level 2 File.

### xskillscore

[**Documentation**][xskillscore-doc] |
[**Sources**][xskillscore-source]

xskillscore is a Python library for computing a wide variety of skill metrics. Its typical application is to verify deterministic and probabilistic forecasts relative to observations.

## Analysis blogposts and tutorials

[Accessing NetCDF and GRIB file collections as cloud-native virtual datasets using Kerchunk, Peter March, Sep 2022](https://medium.com/pangeo/accessing-netcdf-and-grib-file-collections-as-cloud-native-virtual-datasets-using-kerchunk-625a2d0a9191)


[kerchunk-doc]: https://fsspec.github.io/kerchunk/
[kerchunk-source]: https://github.com/fsspec/kerchunk
[APP4-doc]: https://github.com/ACCESS-Hive/APP4
[APP4-source]: https://github.com/ACCESS-Hive/APP4
[ACCESS-Archiver-doc]: https://github.com/ACCESS-Hive/ACCESS-Archiver
[ACCESS-Archiver-source]: https://github.com/ACCESS-Hive/ACCESS-Archiver
[xskillscore-doc]: https://xskillscore.readthedocs.io/en/stable/
[xskillscore-source]: https://github.com/xarray-contrib/xskillscore
[fluxnetlsm-source]: https://github.com/aukkola/FluxnetLSM
[fluxnetlsm-cite]: https://gmd.copernicus.org/articles/10/3379/2017/
[metpy_docu]: https://unidata.github.io/MetPy/latest/examples/formats/index.html
[metpy_docu-source]: https://github.com/Unidata/MetPy
[cmor3-doc]: https://cmor.llnl.gov/
[cmor3-source]: https://github.com/PCMDI/cmor
[xmip-doc]: https://cmip6-preprocessing.readthedocs.io/en/latest/?badge=latest
[xmip-source]: https://github.com/jbusecke/xMIP
[xmip-tutorial]: https://github.com/coecms/xmip_nci
???+ danger "Support Level: NOT supported by ACCESS-NRI"
Here, we collate lists of useful tools for Model Evaluation and Diagnostics (MED). Contrary to the supported content of our [ACCESS-NRI Model Evaluation pages](../../model_evaluation/index.md), the information below is not supported by ACCESS-NRI unless stated otherwise.

Please let us know, if we are missing an important tool. See [How To Contribute](../../contribute/index.md) to get in touch.

<table>
<tr>
<td width="25%">
<div align='center' width="100%">
<h3>Dataset</h3>
</div>
</td>
<td width="75%">
<div align='center' width="100%" >
<h3>Description</h3>
</div>
</td>
</tr>

<tr>
<td width="25%">
<div align='center' width="100%">
<a href="https://cmor.llnl.gov/" target="_blank">CMOR3<br>(Climate Model Output Rewriter Version 3)</a>
</div>
</td>
<td width="75%">
<div align='center' width="100%" >
CMOR is used to produce CF-compliant netCDF files. The structure of the files created by CMOR and the metadata they contain fulfill the requirements of many of the climate community’s standard model experiments (which are referred to here as “MIPs” and include, for example, AMIP, PMIP, APE, and IPCC scenario runs).
<br>
<a href="https://cmor.llnl.gov/" target="_blank">Documentation</a> |
<a href="https://github.com/PCMDI/cmor" target="_blank">Source Code</a>
</div>
</td>
</tr>

<tr>
<td width="25%">
<div align='center' width="100%">
<a href="https://github.com/ACCESS-Hive/APP4" target="_blank">APP4 (The ACCESS Post Processor)</a>
</div>
</td>
<td width="75%">
<div align='center' width="100%" >
The APP4 is a CMORisation tool designed to convert ACCESS model output to ESGF-compliant formats, primarily for publication to CMIP6. The code was originally built for CMIP5, and was further developed for CMIP6-era activities. It uses CMOR3 and files created with the CMIP6 data request to generate CF-compliant files according to the CMIP6 data standards.
<br>
<a href="https://github.com/ACCESS-Hive/APP4" target="_blank">Documentation</a> |
<a href="https://github.com/ACCESS-Hive/APP4" target="_blank">Source Code</a>
</div>
</td>
</tr>

<tr>
<td width="25%">
<div align='center' width="100%">
<a href="https://github.com/ACCESS-Hive/ACCESS-Archiver" target="_blank">ACCESS-Archiver</a>
</div>
</td>
<td width="75%">
<div align='center' width="100%" >
The ACCESS Archiver is designed to archive model output from ACCESS simulations. It's focus is to copy ACCESS model output from its initial location to a secondary location (typically from `/scratch` to `/g/data`), while converting UM files to netCDF, compressing MOM/CICE files, and culling restart files to 10-yearly. Saves 50-80% of storage space due to conversion and compression.
<br>
<a href="https://github.com/ACCESS-Hive/ACCESS-Archiver" target="_blank">Documentation</a> |
<a href="https://github.com/ACCESS-Hive/ACCESS-Archiver" target="_blank">Source Code</a>
</div>
</td>
</tr>

<tr>
<td width="25%">
<div align='center' width="100%">
<a href="https://fsspec.github.io/kerchunk/" target="_blank">Kerchunk</a>
</div>
</td>
<td width="75%">
<div align='center' width="100%" >
Kerchunk is a library that provides a unified way to represent a variety of chunked, compressed data formats (e.g. NetCDF/HDF5, GRIB2, TIFF, …), allowing efficient access to the data from traditional file systems or cloud object storage. It also provides a flexible way to create virtual datasets from multiple files.
Read this <a href="https://medium.com/pangeo/accessing-netcdf-and-grib-file-collections-as-cloud-native-virtual-datasets-using-kerchunk-625a2d0a9191" target="_blank">blogpost</a> on how to access NetCDF and GRIB file colletions with Kerchunk.
<br>
<a href="https://fsspec.github.io/kerchunk/" target="_blank">Documentation</a> |
<a href="https://github.com/fsspec/kerchunk" target="_blank">Source Code</a>
</div>
</td>
</tr>

<tr>
<td width="25%">
<div align='center' width="100%">
<a href="https://cmip6-preprocessing.readthedocs.io/en/latest/?badge=latest" target="_blank">xMIP</a>
</div>
</td>
<td width="75%">
<div align='center' width="100%" >
This package facilitates the cleaning, organization and interactive analysis of Model Intercomparison Projects (MIPs) within the Pangeo software stack.
<br>
<a href="https://cmip6-preprocessing.readthedocs.io/en/latest/?badge=latest" target="_blank">Documentation</a> |
<a href="https://github.com/jbusecke/xMIP" target="_blank">Source Code</a> |
<a href="https://github.com/coecms/xmip_nci" target="_blank">Tutorial</a>
</div>
</td>
</tr>

<tr>
<td width="25%">
<div align='center' width="100%">
<a href="https://espri-mod.github.io/synda/" target="_blank">esgpull and synda</a>
</div>
</td>
<td width="75%">
<div align='center' width="100%" >
esgpull and synda are command line tools to search and download files from the Earth System Grid Federation (ESGF) archive. esgpull is a tool that simplifies usage of the ESGF Search API for data discovery, and manages procedures related to downloading and storing files from ESGF.
<br>
<a href="https://github.com/ESGF/esgf-download" target="_blank">Documentation</a> |
<a href="https://github.com/ESGF/esgf-download" target="_blank">Source Code (esgpull)</a> |
<a href="https://espri-mod.github.io/synda/" target="_blank">Source Code (synda)</a>
</div>
</td>
</tr>

<tr>
<td width="25%">
<div align='center' width="100%">
<a href="https://github.com/aukkola/FluxnetLSM" target="_blank">FluxnetLSM</a>
</div>
</td>
<td width="75%">
<div align='center' width="100%" >
R package for post-processing FLUXNET datasets for use in land surface modelling. Performs quality control and data conversion of FLUXNET data and collated site metadata. Supports FLUXNET2015, La Thuile, OzFlux and ICOS data releases.
<br>
<a href="https://gmd.copernicus.org/articles/10/3379/2017/" target="_blank">Documentation</a> |
<a href="https://github.com/aukkola/FluxnetLSM" target="_blank">Source Code</a>
</div>
</td>
</tr>

<tr>
<td width="25%">
<div align='center' width="100%">
<a href="https://unidata.github.io/MetPy/latest/examples/formats/index.html" target="_blank">MetPy</a>
</div>
</td>
<td width="75%">
<div align='center' width="100%" >
MetPy is a collection of tools in Python for reading, visualizing, and performing calculations with weather data. Format types are GINI Water Vapor Imagery, NEXRAD Level 3 File, and NEXRAD Level 2 File.
<br>
<a href="https://unidata.github.io/MetPy/latest/examples/formats/index.html" target="_blank">Documentation</a> |
<a href="https://github.com/Unidata/MetPy" target="_blank">Source Code</a>
</div>
</td>
</tr>

<tr>
<td width="25%">
<div align='center' width="100%">
<a href="https://xskillscore.readthedocs.io/en/stable/" target="_blank">xskillscore</a>
</div>
</td>
<td width="75%">
<div align='center' width="100%" >
xskillscore is a Python library for computing a wide variety of skill metrics. Its typical application is to verify deterministic and probabilistic forecasts relative to observations.
<br>
<a href="https://xskillscore.readthedocs.io/en/stable/" target="_blank">Documentation</a> |
<a href="https://github.com/xarray-contrib/xskillscore" target="_blank">Source Code</a>
</div>
</td>
</tr>
</table>
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Community Model Evaluation and Diagnostics (MED) Recipe Gallery

<!-- {% include "call_contribute.md" %} -->
???+ danger "Support Level: NOT supported by ACCESS-NRI"
Here, we collate lists of useful resources for Model Evaluation and Diagnostics (MED). Contrary to the supported content of our [ACCESS-NRI Model Evaluation pages](../../model_evaluation/index.md), the information below is not supported by ACCESS-NRI unless stated otherwise.

We are trying to ingest more and more model evaluation and diagnostics recipes in your currated [recipe gallery on this website](../../model_evaluation/model_evaluation_recipe_gallery.md). While this is a continous effort, this site is intented for a list of model evaluation and diagnostics recipes that are not (yet) ingested but may be interesting for the community:

Expand Down Expand Up @@ -379,4 +380,4 @@ IOMB uses the same code base as the International Land Model Benchmarking (ILAMB
</tr>


<table/>
</table>
Loading

0 comments on commit a466110

Please sign in to comment.