On the use of NetCDF to run ai-models

[aperaza-bsc]

Hey all,

I am trying to run ai-models on a set of models. I want to launch multiple forecasts at different timesteps and compare them with the outputs of ERA5. Due to the requirements of my HPC, I do not have access to internet and thus, needed to download the data. For that purpose, I have created a Zarr dataset from netcdf previously requested from CDS. From this dataset I extract the initial conditions required for a model and convert them to netcdf to use as input for ai-models.

Now, I'm encountering many difficulties mainly on the use of earthkit-data to parse this netcdf files. The main one is on the use of the object NetCDFMultiFieldList as it's not implementing to_xarray.
To guide why this object is created, function reader from earthkit.data.readers.netcdf returns fs as fs.has_fields() returns True. This creates a NetCDFFieldListReader with all the fields read from the netcdf. Afterwards, this object mutates to a NetCDFMultiFieldList and ai-models tries to convert it to Xarray and fails since this conversion is not implemented.
If fs.has_fields() were to be False, a NetCDFReader object would be created, while this raises other errors, a conversion to Xarray is implemented for it. I have noticed that the methods mutate_source from NetCDFFieldListReader and NetCDFFieldListUrlReader have comments regarding themselves as already being NetCDFReader objects (" # A NetCDFReader is a source itself"). I wonder if these classes were supposed to inherit from NetCDFReader or if there are plans for this to occur at some point.

I will also comment this issue in the earthkit-data repository.

Thanks,
Alejandro

[HCookie]

Hi,

As Sandor pointed out in the earthkit.data repo ai-models has no current support for non grib data.
This is in part due to metadata checking when saving fields, as well as the inability for netcdf / xarray objects to be saved to grib.
The direct error message you recieve above is in part resolved by the newer release of ai-models.
But you will encounter more.
Firstly, you will need to rename the surface vars
.rename(u10 = '10u', v10 = '10v', t2m='2t')

Finally, I have just pushed to a new branch to provide experimental support for xarray objects.
Please try it out, and let me know if it works. https://github.com/ecmwf-lab/ai-models/tree/feature/netcdf-support
ai-models --download-assets --input file --file pangu_test.nc --output netcdf --path pangu_output.nc --date 20230101 --time 1200 --lead-time 12 panguweather