Skip to content

Commit

Permalink
adjustments to model evaluation
Browse files Browse the repository at this point in the history
  • Loading branch information
@svenbuder
svenbuder committed Jul 13, 2023
1 parent 16cf86b commit 871c03f
Show file tree
Hide file tree
Showing 22 changed files with 171 additions and 25,282 deletions.
Binary file added docs/assets/model_evaluation/condaenv_list.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/model_evaluation/condaenv_raise_issue.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/model_evaluation/logo_cosima.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
19 changes: 19 additions & 0 deletions docs/community_resources/community_med_recipes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -360,4 +360,23 @@ IOMB uses the same code base as the International Land Model Benchmarking (ILAMB
</td>
</tr>


<tr>
<td width="25%">
<div align='center' width="100%">
<a href="https://cookbooks.projectpythia.org">Pythia Cookbooks {{ community }}</a>
</div>
</td>
<td width="10%">
</td>
<td width="64%">
<div align='center' width="100%" >
<a href="https://cookbooks.projectpythia.org">Documentation</a> |
<a href="https://github.com/ProjectPythia">Source </a>
<!-- The Afterburner project is a multi-year initiative of the UK Met Office to develop a suite of robust, reusable scientific software tools and applications which can be integrated into climate model runs to deliver commonly needed in-line (as a model is running) and post-processing functionality. The project is funded by the UK government through a combination of direct grant and the ​[Newton Fund](http://www.newtonfund.ac.uk). -->
</div>
</td>
</tr>


<table/>
67 changes: 39 additions & 28 deletions docs/model_evaluation/index.md
View file
Original file line number Diff line number Diff line change
@@ -1,80 +1,99 @@
# <div class="highlight-bg"> Model Evaluation and Diagnostics (MED) </div>

Model evaluation is about measuring how fit for purpose a particular model is. If you are new to model evaluation and diagnostics, we recommend you read our [Getting Started with MED page](./model_evaluation_getting_started/index.md). Here, we provide catalogs and pointers to [observational data](./model_evaluation_observational_catalogs.md) as well as [model data](./model_evaluation_model_catalogs/index.md) that can be used for evaluation. We provide [tools to process such data](./model_evaluation_data_processing.md) into a comparable format and a [gallery of recipes to evaluate](https://medportal.herokuapp.com/models/published) the formatted data.
<!-- Model evaluation is about measuring how fit for purpose a particular model is. -->

## Getting Started
## What is MED about?

ACCESS-NRI's "Model Evaluation and Diagnostics" work is a critical facet of climate modeling, encompassing various tasks designed to ensure the model's reliability and accuracy.

**Evaluation** involves scrutinizing the model through Model/Observation confrontations, checking its performance against real-world observations. It also includes experiment comparisons, testing the model under different scenarios, and inter-model comparisons like the Coupled Model Intercomparison Project (CMIP), assessing how the ACCESS-NRI model fares when compared with other climate models.

**Diagnostics** involves constant monitoring of model runs to detect any anomalies or inconsistencies and a thorough analysis of outputs to verify the model's accuracy over time.

## ACCESS-MED data and tools hosted on Gadi

If you are new to model evaluation and diagnostics, we recommend you read our [Getting Started with MED page](./model_evaluation_getting_started/index.md). Here, we provide catalogs and pointers to [observational data](./model_evaluation_observational_catalogs.md) as well as [model data](./model_evaluation_model_catalogs/index.md) that can be used for evaluation. We also provide a number of [frameworks for model evaluation](./model_evaluation_on_gadi/index.md). We are also working on implementing more frameworks and recipes as well as formatting tools for a better model evaluation and diagnostics.

### Getting Started
<div class="card-container">
<a href="./model_evaluation_getting_started/access_to_gadi_at_nci.md" class="squared-card default-text-color">
<a href="./model_evaluation_getting_started/access_to_gadi_at_nci" class="squared-card default-text-color">
<div class="squared-card-image">
<img src="..//assets/model_evaluation/Gadi-19-2.jpg" alt="Computing Access"></img>
</div>
<div class="squared-card-text bold">Computing Access</div>
</a>
<a href="./model_evaluation_getting_started/model_evaluation_getting_started.md" class="squared-card default-text-color">
<a href="./model_evaluation_getting_started/model_evaluation_getting_started" class="squared-card default-text-color">
<div class="squared-card-image">
<img src="../assets/model_evaluation/model_evaluation_conda.png" alt="MED Conda Environment"></img>
</div>
<div class="squared-card-text bold">MED Conda Environment</div>
</a>
<a href="./model_evaluation_getting_started/model_variables/index.md" class="squared-card default-text-color">
<a href="./model_evaluation_getting_started/model_variables" class="squared-card default-text-color">
<div class="squared-card-image">
<img src="../assets/model_evaluation/model_evaluation_variables.png" alt="Model Variables"></img>
</div>
<div class="squared-card-text bold">Model Variables</div>
</a>
</div>

## Data Catalogs
### Data Catalogs
<div class="card-container">
<a href="./model_evaluation_observational_catalogs.md" class="squared-card default-text-color">
<a href="./model_evaluation_observational_catalogs" class="squared-card default-text-color">
<div class="squared-card-image">
<img src="../assets/model_evaluation/model_evaluation_obs_catalog.jpg" alt="A picture of a seismograph recording seismic waves during an earthquake visualises the link to our Observational Data Catalog. Image credit: Wf Sihardian—EyeEm/Getty Images" title="Image credit: Wf Sihardian—EyeEm/Getty Images"></img>
</div>
<div class="squared-card-text bold">Observational Data Catalog</div>
</a>
<a href="./model_evaluation_model_catalogs/index.md" class="squared-card default-text-color">
<a href="./model_evaluation_model_catalogs" class="squared-card default-text-color">
<div class="squared-card-image">
<img src="../assets/model_evaluation/model_evaluation_model_catalog.jpg" alt="MED Conda Environment"></img>
</div>
<div class="squared-card-text bold">Model Data Catalog</div>
</a>
</div>

## Evaluation Tools

### Frameworks on Gadi
### Supported Community Frameworks on Gadi

<div class="card-container">
<a href="./model_evaluation_getting_started/access_to_gadi_at_nci.md" class="squared-card default-text-color">
<a href="./model_evaluation_on_gadi/model_evaluation_on_gadi_ilamb" class="squared-card default-text-color">
<div class="squared-card-image">
<img src="../assets/model_evaluation/logo_ilamb.png" alt="ILAMB"></img>
</div>
<div class="squared-card-text bold">ILAMB</div>
</a>
<a href="./model_evaluation_getting_started/model_evaluation_getting_started.md" class="squared-card default-text-color">
<a href="./model_evaluation_on_gadi/model_evaluation_on_gadi_esmvaltool" class="squared-card default-text-color">
<div class="squared-card-image">
<img src="../assets/model_evaluation/logo_esmvaltool.png" alt="ESMValTool"></img>
</div>
<div class="squared-card-text bold">ESMValTool</div>
</a>
<a href="./model_evaluation_getting_started/model_variables/index.md" class="squared-card default-text-color">
<a href="./model_evaluation_on_gadi/model_evaluation_on_gadi_pangeo_cosima" class="squared-card default-text-color">
<div class="squared-card-image">
<img src="../assets/model_evaluation/logo_pangeo.png" alt="Pangeo/COSIMA"></img>
<img src="../assets/model_evaluation/logo_cosima.png" alt="Pangeo/COSIMA"></img>
</div>
<div class="squared-card-text bold">Pangeo/COSIMA</div>
<div class="squared-card-text bold">COSIMA cookbook</div>
</a>
</div>

### What is to come: Diagnostics, Recipe Gallery and more
### Tools in development

We are currently setting up a range of tools that will help you to better evaluate and diagnose climate models:
* Model Diagnostics for on-the-fly analysis of your models at different snapshots
* Data format processing tools like APP4
* An Evaluation Recipe Gallery with searching functionality

While we are working on these, we have collected a number of links to existing tools in our [community tab](../community_resources/index.md) (note that we are not currating them).

{% include "call_contribute.md" %}

<!--
<div class="card-container">
<a href="./model_evaluation_getting_started/access_to_gadi_at_nci.md" class="squared-card default-text-color">
<div class="squared-card-image">
<img src="..//assets/model_evaluation/Gadi-19-2.jpg" alt="Computing Access"></img>
<img src="..//assets/model_evaluation/Gadi-19-2.jpg" alt="Model Diagnostics"></img>
</div>
<div class="squared-card-text bold">Computing Access</div>
<div class="squared-card-text bold">Model Diagnostics</div>
</a>
<a href="./model_evaluation_data_processing.md" class="squared-card default-text-color">
<div class="squared-card-image">
Expand All @@ -88,19 +107,11 @@ Model evaluation is about measuring how fit for purpose a particular model is. I
</div>
<div class="squared-card-text bold">Evaluation Recipe Gallery</div>
</a>
</div>
</div> -->


<!-- THIS NEEDS TO BE DONE
## TBD: FRAMEWORKS AND RECIPES
TBD: ILAMB, ESMVALTOOL, COSMIA -> Pangeo (extended COSIMA)
## TBD: COMMUNITY MED
TBD: Mention Community MED in our curated MED
## TBD: CMORisation
TBD: Raw data vs. curated data: CMORized vs. not! What does CMORized actually mean (look at ESMValTool documentation)?
Expand Down
22 changes: 20 additions & 2 deletions docs/model_evaluation/model_evaluation_getting_started/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,23 @@ Welcome to Model Evaluation and Diagnostics!

Here, we provide you the important information to give you access to the large data that we curate at NCI's storage and show you how you can use it to figure out how fit for purpose specific models are, in particular when you compare them to osbervational data:

1) [Getting Access to NCI and relevant NCI projects](./access_to_gadi_at_nci.md)
2) [Setting up environments at gadi@NCI to load and evaluate observational and model data](./model_evaluation_getting_started.md)
<div class="card-container">
<a href="./access_to_gadi_at_nci" class="squared-card default-text-color">
<div class="squared-card-image">
<img src="../../assets/model_evaluation/Gadi-19-2.jpg" alt="Computing Access"></img>
</div>
<div class="squared-card-text bold">Computing Access</div>
</a>
<a href="./model_evaluation_getting_started" class="squared-card default-text-color">
<div class="squared-card-image">
<img src="../../assets/model_evaluation/model_evaluation_conda.png" alt="MED Conda Environment"></img>
</div>
<div class="squared-card-text bold">MED Conda Environment</div>
</a>
<a href="./model_variables" class="squared-card default-text-color">
<div class="squared-card-image">
<img src="../../assets/model_evaluation/model_evaluation_variables.png" alt="Model Variables"></img>
</div>
<div class="squared-card-text bold">Model Variables</div>
</a>
</div>
39 changes: 11 additions & 28 deletions ...evaluation/model_evaluation_getting_started/model_evaluation_getting_started.md
View file
Original file line number Diff line number Diff line change
@@ -1,49 +1,32 @@
# Getting Started with Model Evaluation at Gadi@NCI
# Model Evaluation Environment at Gadi@NCI

At this stage of *Getting Started*, we assume that you already have access to Gadi@NCI. If this is not the case, please go to our instructions on how to get [access to Gadi@NCI](./access_to_gadi_at_nci.md)
At this stage of *Getting Started*, we assume that you already have access to Gadi@NCI. If this is not the case, please go to our instructions on how to get [access to Gadi@NCI](./access_to_gadi_at_nci.md).

Here we describe where you can find, load, and evalulate observational and model data on Gadi.

Note: You do not automatically have access to all of Gadi's storage at `/g/data/`, but need to be part of a `$PROJECT` to see files at `/g/data/$PROJECT`. Furthermore, if you use Gadi's job submission system PBS (Portable Batch System), you need to add the relevant storage to the `#PBS -l storage=gdata/xp65+gdata/kj13` (if you want the job to have access to `xp65` and `kj13` in this example).
???+ warning "ACCESS-NRI provides code and data, but not computing resources"
You do not automatically have access to all of Gadi's storage at `/g/data/`, but need to be part of a `$PROJECT` to see files at `/g/data/$PROJECT`. Furthermore, if you use Gadi's job submission system PBS (Portable Batch System), you need to add the relevant storage to the `#PBS -l storage=gdata/xp65+gdata/kj13` (if you want the job to have access to `xp65` and `kj13` in this example).

### 2.1) `access-med`: Our currated `conda` environment for you on Gadi
## 1 `access-med`: Our currated `conda` environment on Gadi

To avoid running multiple (different) versions of code on Gadi, we provide you with a `conda` environment called `access-med` that we actually curate for you (version 0.1 is from June 2023).

In order to change to this environment, please execute the following commands after loggin onto Gadi (and as part of your PBS scripts):
<terminal-animation>
<terminal-line data="input">module use /g/data/xp65/public/modules</terminal-line>
<terminal-line data="input">module load conda/access-med</terminal-line>
<terminal-line>Loading conda/access-med-0.1</terminal-line>
<terminal-line> Loading requirement: singularity</terminal-line>
<terminal-line data="input">esmvaltool recipes list</terminal-line>
</terminal-animation>

```
$ module use /g/data/xp65/public/modules
$ module load conda/access-med
```

If you are planning to run your code through JupyterLab on [NCI's ARE](https://are.nci.org.au), you need to use `/g/data/xp65/public/modules` as **Module directories** and `conda/are` as **Modules** when launching a JupyterLab session.

You are now able to use the scripts of our currated environment, including `python3`, `intake`, `jupyter`, `esmvaltool`, or `ilamb`. The complete list of dependencies can be found in our dedicated [GitHub repository](https://github.com/ACCESS-NRI/MED-condaenv/blob/main/scripts/environment.yml).

### 2.2) Observational Data

We provide an extensive collection of observational data on Gadi@NCI within the `/g/data/kj13/datasets` directory.

Please take a look at our [Observational Data Catalog](../model_evaluation_observational_catalogs.md) for an overview.

### 2.3) Model Data

There are many models and data stored on Gadi, as you can imagine from the plethora of projects in [Section 1.2](#12-join-relevant-nci-projects). Downloading this data is hardly practical, so we suggest to work on Gadi instead.
## 2 What is part of the `access-med` enrivonment?

To avoid endless searches within Gadi's storage, we written a useful 'library' tool, called `access-nri-catalog`, that allows to search the Model Catalogs easily and is already loaded as part of our `access-med` `conda` environment. To find out how you can search for Model Data on Gadi, take a look at our [Model Catalog](../model_evaluation_model_catalogs/model_evaluation_search_models.md).
You are now able to use the scripts of our currated environment, including `python3`, `intake`, `jupyter`, `esmvaltool`, or `ilamb`. The complete list of dependencies can be found in the `environment.yml` file of our [GitHub repository](https://github.com/ACCESS-NRI/MED-condaenv/blob/main/scripts/environment.yml):

### 2.5) Model Evaluation
[![Raise Issue](../../assets/model_evaluation/condaenv_list.png)](https://github.com/ACCESS-NRI/MED-condaenv/issues)

Now that you have both [Observational Data](#22-observational-data) and [Model Data](#23-model-data) at the palm of your hand on Gadi@NCI, there are many ways to evaluate this data.
## 3 If you need more packages

As part of our ACCESS-NRI `conda` environment, we provide several Model Evaluation Tools, like `ilamb` or `esmvaltool`.
Simply raise an issue to *add a package* in our [ACCESS-NRI/MED-condaenv GitHub repository](https://github.com/ACCESS-NRI/MED-condaenv/issues):

Check out [Model Evaluation at Gadi](../model_evaluation_on_gadi/index.md) to find out how you can use them on Gadi.
[![Raise Issue](../../assets/model_evaluation/condaenv_raise_issue.png)](https://github.com/ACCESS-NRI/MED-condaenv/issues)
Loading

0 comments on commit 871c03f

Please sign in to comment.