Fixed provenance logging of all python diagnostics by removing 'plot_file' entry

[schlunma]

Update the provenance logging of all python diagnostics by removing the plot_file attribute.

• Related to Fix provenance of NCL figures created using the log_provenance function #2279 and Work around for broken provenance of some Python diagnostics #2288

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🛠 This pull request has a descriptive title
• 🛠 Code is written according to the code quality guidelines
• 🛠 Documentation is available
• 🛠 Tests run successfully
• 🛠 The list of authors is up to date
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 All checks below this pull request were successful

---

To help with the number of pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[schlunma]

@esmvalbot Please run recipe_cox18nature.yml

[esmvalbot]

Since @schlunma asked, ESMValBot will run recipe recipe_cox18nature.yml as soon as possible, output will be generated here

[schlunma]

@esmvalbot Please run recipe_tcr.yml

[esmvalbot]

Since @schlunma asked, ESMValBot will run recipe recipe_tcr.yml as soon as possible, output will be generated here

[esmvalbot]

ESMValBot is happy to report recipe recipe_cox18nature.yml ran OK, output has been generated here

[esmvalbot]

ESMValBot is happy to report recipe recipe_tcr.yml ran OK, output has been generated here

[schlunma]

@esmvalbot Please run recipe_autoassess_landsurface_soilmoisture.yml

[esmvalbot]

Since @schlunma asked, ESMValBot will run recipe recipe_autoassess_landsurface_soilmoisture.yml as soon as possible, output will be generated here

[esmvalbot]

ESMValBot is sorry to report it failed to run recipe recipe_autoassess_landsurface_soilmoisture.yml: exit is 1, output has been generated here

[schlunma]

I don't think the fail of the recipe is related to changes here... @valeriupredoi could you check, please?

[bouweandela]

That looks like the first problem I reported in this list: #2198 (comment). @valeriupredoi Any chance of making that auxiliary data available in a public repository (e.g. Zenodo)?

[valeriupredoi]

@schlunma nein, the fails is due to what @bouweandela says - missing ancil/mask file; the problem with those files is that they're not small, so not sure where'd be the bast place to put them. Here are sizes:

-rw-rw---- 1 valeriu gws_esmeval    3140420 Feb 20  2020 CERES-EBAF_Ed2-6r_200003-201002.ann.nc
-rw-rw---- 1 valeriu gws_esmeval    3140316 Feb 20  2020 CERES-EBAF_Ed2-6r_200003-201002.djf.nc
-rw-rw---- 1 valeriu gws_esmeval    3140316 Feb 20  2020 CERES-EBAF_Ed2-6r_200003-201002.jja.nc
-rw-rw---- 1 valeriu gws_esmeval    3140316 Feb 20  2020 CERES-EBAF_Ed2-6r_200003-201002.mam.nc
-rw-rw---- 1 valeriu gws_esmeval    3140316 Feb 20  2020 CERES-EBAF_Ed2-6r_200003-201002.son.nc
drwxrwsr-x 1 valeriu gws_esmeval          0 Feb 20  2020 clisccp
drwxrwsr-x 1 valeriu gws_esmeval          0 Feb 20  2020 crutem3
-rw-rw---- 1 valeriu gws_esmeval    4166393 Feb 20  2020 ecv_soil_moisture_djf.nc
-rw-rw---- 1 valeriu gws_esmeval    4166393 Feb 20  2020 ecv_soil_moisture_jja.nc
-rw-rw---- 1 valeriu gws_esmeval    4166393 Feb 20  2020 ecv_soil_moisture_mam.nc
-rw-rw---- 1 valeriu gws_esmeval    4166393 Feb 20  2020 ecv_soil_moisture_son.nc
drwxrwsr-x 1 valeriu gws_esmeval          0 Feb 20  2020 ERAI
drwxrwsr-x 1 valeriu gws_esmeval          0 Feb 20  2020 GPCP
drwxrwsr-x 1 valeriu gws_esmeval          0 Feb 20  2020 HadGHCND
drwxrwsr-x 1 valeriu gws_esmeval          0 Feb 20  2020 HadSLP2
drwxrwsr-x 1 valeriu gws_esmeval          0 Feb 20  2020 MAR_3.5.2
-rw-rw---- 1 valeriu gws_esmeval 1469687408 Feb 20  2020 MAR_ref_values.tar.gz
-rw-rw---- 1 valeriu gws_esmeval     111632 Feb 20  2020 mask_frac_v9_n96.pp
drwxrwsr-x 1 valeriu gws_esmeval          0 Feb 20  2020 masks
-rw-rw---- 1 valeriu gws_esmeval    3252508 Feb 20  2020 obs_BHR_NIR_seasons_1998-2008.nc
-rw-rw---- 1 valeriu gws_esmeval    3252508 Feb 20  2020 obs_BHR_VIS_seasons_1998-2008.nc
-rw-rw---- 1 valeriu gws_esmeval    3252508 Feb 20  2020 obs_DHR_NIR_seasons_1998-2008.nc
-rw-rw---- 1 valeriu gws_esmeval    3252508 Feb 20  2020 obs_DHR_VIS_seasons_1998-2008.nc
drwxrwsr-x 1 valeriu gws_esmeval          0 Feb 20  2020 RACMO
-rw-rw---- 1 valeriu gws_esmeval  484336986 Feb 20  2020 RACMO_ref_values.tar.gz
drwxrwsr-x 1 valeriu gws_esmeval          0 Feb 20  2020 SSMI
-rw-rw---- 1 valeriu gws_esmeval    3545396 Feb 20  2020 SWE_clm_djf.pp
-rw-rw---- 1 valeriu gws_esmeval    3545396 Feb 20  2020 SWE_clm_mam.pp
-rw-rw---- 1 valeriu gws_esmeval    3545396 Feb 20  2020 SWE_clm_son.pp

[bouweandela]

Just upload them to Zenodo:

What are the size limits in Zenodo?

We currently accept up to 50GB per dataset (you can have multiple datasets); there is no size limit on communities. However, we don't want to turn away larger use cases. If you would like to upload larger files, please contact us, and we will do our best to help you. Please be aware that we cannot offer infinite space for free, so donations towards sustainability are encouraged.

from https://help.zenodo.org/

[bouweandela]

What about those 0 bytes files?

[valeriupredoi]

sounds good! I'll have to figure out how to move files from JASMIN to Zenodo 😁 The 0 size ones are dirs, not sure we need thos at all, gotta check...

[bouweandela]

Something like this? https://gist.github.com/maxogden/b758cf0fe6d353846ef9ce7d03fdca0c

[zklaus]

I worry a bit that the scope of this PR is creeping out of control. Could it make sense to merge this as is and relegate zenodo and other things to other issues/PRs?

[schlunma]

I worry a bit that the scope of this PR is creeping out of control. Could it make sense to merge this as is and relegate zenodo and other things to other issues/PRs?

I agree. The scope of this PR is to remove the plot_file attribute from the python diagnostics. It would be nice if we could merge this soon - I will be on vacation starting next week and won't be back in time for the feature freeze.

[valeriupredoi]

I will upload those files to Zenodo eventually, first I need to make sure I am allowed to do that, they are Met Office property so sharing them via Zenodo is pending their approval; that might take a while so let's not make that the roadblock for this PR. @bouweandela those 0-size files are dirs; I'll have a quick look now at the actual code changes in the Autoassess recipes, then if all's good will approve 👍

[valeriupredoi]

OK I ran all the Autoassess recipes succesfully on JASMIN, all good by me, thanks Manu! I will go ahead and open an issue about uploading those ancil files to Zenodo now 👍

[bouweandela]

Let's get this merged then!