Data download failure for recipe_easy_ipcc.yml

[ehogan]

Describe the bug
Following ESMValCore v2.11.0rc2 testing, the recipe_easy_ipcc.yml recipe is failure with many occurrences of the following error:

2024-06-27 18:18:17,925 UTC [917288] ERROR   Failed to download 
ESGFFile:CMIP6/ScenarioMIP/AS-RCEC/TaiESM1/ssp585/r1i1p1f1/Omon/tos/gn/v20240426/tos_Omon_TaiESM1_ssp585_r1i1p1f1_gn_208901-208912.nc
on hosts ['esgf.rcec.sinica.edu.tw'], error message Failed to download file
/work/bd0854/DATA/ESMValTool2/download/CMIP6/ScenarioMIP/AS-RCEC/TaiESM1/ssp585/r1i1p1f1/Omon/tos/gn/v20240426/tos_Omon_TaiESM1_ssp585_r1i1p1f1_gn_208901-208912.nc,
errors:
http://esgf.rcec.sinica.edu.tw/thredds/fileServer/my_cmip6_dataroot/ScenarioMIP/AS-RCEC/TaiESM1/ssp585/r1i1p1f1/Omon/tos/gn/v20240426/tos_Omon_TaiESM1_ssp585_r1i1p1f1_gn_208901-208912.nc:
404 Client Error: 404 for url:
http://esgf.rcec.sinica.edu.tw/thredds/fileServer/my_cmip6_dataroot/ScenarioMIP/AS-RCEC/TaiESM1/ssp585/r1i1p1f1/Omon/tos/gn/v20240426/tos_Omon_TaiESM1_ssp585_r1i1p1f1_gn_208901-208912.nc

This recipe failed in the same way during ESMValCore v2.11.0rc1 testing. The recipe worked in ESMValTool v2.10.0.

[bouweandela]

Is it only the TaiESM1 dataset that fails to download?

[ehogan]

On closer inspection there are also entries like this in the log:

2024-06-27 18:20:21,604 UTC [917288] ERROR   Failed to download 
ESGFFile:CMIP6/ScenarioMIP/CSIRO/ACCESS-ESM1-5/ssp126/r40i1p1f1/Omon/tos/gn/v20240509/tos_Omon_ACCESS-ESM1-5_ssp126_r40i1p1f1_gn_201501-210012.nc
on hosts ['esgf.nci.org.au'], error message Failed to download file
/work/bd0854/DATA/ESMValTool2/download/CMIP6/ScenarioMIP/CSIRO/ACCESS-ESM1-5/ssp126/r40i1p1f1/Omon/tos/gn/v20240509/tos_Omon_ACCESS-ESM1-5_ssp126_r40i1p1f1_gn_201501-210012.nc,
errors:
http://esgf.nci.org.au/thredds/fileServer/master/CMIP6/ScenarioMIP/CSIRO/ACCESS-ESM1-5/ssp126/r40i1p1f1/Omon/tos/gn/v20240509/tos_Omon_ACCESS-ESM1-5_ssp126_r40i1p1f1_gn_201501-210012.nc:
('Connection broken: IncompleteRead(42715483 bytes read, 213683214 more expected)', 
IncompleteRead(42715483 bytes read, 213683214 more expected))

Other than that they are all TaiESM1 errors.

[bouweandela]

Let me take the TaiESM1 dataset out then, I'll make a pull request. The ACCESS-ESM1-5 error looks like an intermittent issue with the internet connection that may go away if you run it again.