[Bug]: Update license attribute in default_metadata.json (invalid with CMOR) -- actually need to update zppy version

[tomvothecoder]

What happened?

The license attribute is considered invalid with CMOR now.

There is a difference between the default_metadata.jsonof e3sm_to_cmip (left) and zppy (right):

I ran e3ms_to_cmip with both versions of that file. The e3sm_to_cmip version fails because of the license attribute being considered invalid with cmor.write, while the zppy version succeeds.

What did you expect to happen? Are there are possible answers you came across?

Update the license attribute to match zppy's version.

Minimal Complete Verifiable Example (MVCE)

from e3sm_to_cmip.__main__ import E3SMtoCMIP

# e3sm_to_cmip --output-path ~/test -v 'lai' --realm lnd --input-path /lcrc/group/e3sm/ac.zhang40/zppy_test_complete_run_output/test-main2-20240216/v2.LR.historical_0201_try7/post/lnd/180x360_aave/ts/monthly/2yr --user-metadata ~/zppy/zppy/templates/e3sm_to_cmip/default_metadata.json --tables-path /lcrc/group/e3sm/diagnostics/cmip6-cmor-tables/Tables --num-proc 12

CMOR_TABLES_DIR = "/lcrc/group/e3sm/diagnostics/cmip6-cmor-tables/Tables"
INPUT_DIR = "/lcrc/group/e3sm/ac.zhang40/zppy_test_complete_run_output/test-main2-20240216/v2.LR.historical_0201_try7/post/lnd/180x360_aave/ts/monthly/2yr"
OUTPUT_PATH = "/home/ac.tvo/E3SM-Project/e3sm_to_cmip/scripts/debug/549_zppy_ts"
METADATA_PATH = "/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/resources/default_metadata.json"

args = [
    "--output-path",
    OUTPUT_PATH,
    "--var-list",
    "lai",
    "--realm",
    "lnd",
    "--input-path",
    INPUT_DIR,
    "--user-metadata",
    METADATA_PATH,
    "--num-proc",
    "12",
    "--tables-path",
    CMOR_TABLES_DIR,
    "--serial",
]

run = E3SMtoCMIP(args)

run.run()

Relevant log output

2024-02-22 19:55:35,155 [INFO]: handler.py(_cmor_write_with_time:680) >> lai: Writing variable to file...
2024-02-22 19:55:35,162 [ERROR]: handler.py(_cmor_write_with_time:690) >> Problem with 'cmor.write'. Please check the logfile (if defined).

�[2;34;47mC Traceback:
In function: _CV_setInstitution
! called from: cmor_setGblAttr
! called from: cmor_write
! �[0m

�[1;34;47m!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Warning: Your input attribute institution "E3SM-Project" will be replaced with 
! "LLNL (Lawrence Livermore National Laboratory, Livermore, CA 94550, USA); ANL (Argonne National Laboratory, Argonne, IL 60439, USA); BNL (Brookhaven National Laboratory, Upton, NY 11973, USA); LANL (Los Alamos National Laboratory, Los Alamos, NM 87545, USA); LBNL (Lawrence Berkeley National Laboratory, Berkeley, CA 94720, USA); ORNL (Oak Ridge National Laboratory, Oak Ridge, TN 37831, USA); PNNL (Pacific Northwest National Laboratory, Richland, WA 99352, USA); SNL (Sandia National Laboratories, Albuquerque, NM 87185, USA). Mailing address: LLNL Climate Program, c/o David C. Bader, Principal Investigator, L-103, 7000 East Avenue, Livermore, CA 94550, USA" as defined in your Control Vocabulary file.
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!�[0m


�[2;31;47mC Traceback:
! In function: _CV_ValidateAttribute
! called from: _CV_checkGblAttributes
! called from: cmor_write
! �[0m

�[1;31;47m!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: The attribute "license" could not be validated. 
! The current input value is "CMIP6 model data produced by E3SM-Project is licensed under a Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and at https://pcmdi.llnl.gov/. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.", which is not valid. 
! 
! Valid values must match those found in the "license" section
! of your 
!
!!!!!!!!!!!!!!!!!!!!!!!!!�[0m


�[2;31;47mC Traceback:
! In function: _CV_checkGblAttributes
! called from: cmor_write
! �[0m

�[1;31;47m!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Please fix required attributes mentioned in
! the warnings/error above and rerun. (aborting!)
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!�[0m


�[2;31;47mC Traceback:
! In function: �[0m

�[1;31;47m!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Cannot continue until you fix the errors listed above: -1
!
!!!!!!!!!!!!!!!!!!!!!!!!!�[0m

Anything else we need to know?

No response

Environment

Latest version of e3sm_to_cmip and master branch

[chengzhuzhang]

@tomvothecoder tracking back the license change: E3SM-Project/CMIP6-Metadata#7
It seems to be the one one the left hand side reflected the updated license information. In the test, we need to ensure the metadata file, cmore tables and cmor version (>=3.7.0) are all up to date...

[chengzhuzhang]

The zppy version is up-to-date, but it is on the left side, as below...

[tomvothecoder]

The zppy version is up-to-date, but it is on the left side, as below...

The left one matches e3sm_to_cmip/resources/default_metadata.json and the right one matches zppy/templates/e3sm_to_cmip/default_metadata.json. I provided direct hyperlinks to both.

I think you're saying e3sm_to_cmip (left) is up-to-date, but that is the one failing with my run script. I am using these paths with the latest dev environment that includes cmor=3.7.3.

CMOR_TABLES_DIR = "/lcrc/group/e3sm/diagnostics/cmip6-cmor-tables/Tables"
INPUT_DIR = "/lcrc/group/e3sm/ac.zhang40/zppy_test_complete_run_output/test-main2-20240216/v2.LR.historical_0201_try7/post/lnd/180x360_aave/ts/monthly/2yr"
OUTPUT_PATH = "/home/ac.tvo/E3SM-Project/e3sm_to_cmip/scripts/debug/549_zppy_ts"

# Fails with this path
METADATA_PATH = "/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/resources/default_metadata.json"

# Succeeds with this path
# METADATA_PATH = "/home/ac.tvo/E3SM-Project/zppy/templates/e3sm_to_cmip/default_metadata.json"

[tomvothecoder]

I'll do a git pull on CMOR_TABLES_DIR = "/lcrc/group/e3sm/diagnostics/cmip6-cmor-tables/Tables" to make sure it is up to date, then re-run with the e3sm_to_cmip version of this default_metadata.json.

EDIT: Nevermind I can't do a git pull on this repo because it is owned by Xylar. I'll try a local copy of the latest version.

[tomvothecoder]

My findings:

1. zppy version of default_metadata.json is outdated -- needs to be updated
2. e3sm_to_cmip version of default_metadata.json is up-to-date
3. "/lcrc/group/e3sm/diagnostics/cmip6-cmor-tables/Tables" is outdated (which is why it works with 1 and not 2) -- needs to be updated

[chengzhuzhang]

Thanks for clarifying. It seems like the timing to update 1, and 3 could be tricky. I suppose we should update 1 first, and then update 3 once e3sm_unified is deployed with latest zppy?

[tomvothecoder]

Thanks for clarifying. It seems like the timing to update 1, and 3 could be tricky. I suppose we should update 1 first, and then update 3 once e3sm_unified is deployed with latest zppy?

We can do this after the next E3SM-Unified release to avoid any additional headaches. 1 and 3 need to be updated at the same time. The latest version of default_metadata.json (2) doesn't work with 3 until it is updated.

This issue can be transferred over to zppy.