problem running ai-models with ecmwf open data

[bakamotokatas]

Hello,

I am getting following error when running

ai-models --input opendata --lead-time 24 graphcast

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/xarray/core/dataset.py", line 1408, in _copy_listed
    variables[name] = self._variables[name]
                      ~~~~~~~~~~~~~~~^^^^^^
KeyError: 'geopotential'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/ai-models", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/lib/python3.11/site-packages/ai_models/__main__.py", line 322, in main
    _main(sys.argv[1:])
  File "/usr/lib/python3.11/site-packages/ai_models/__main__.py", line 270, in _main
    run(vars(args), unknownargs)
  File "/usr/lib/python3.11/site-packages/ai_models/__main__.py", line 295, in run
    model.run()
  File "/usr/lib/python3.11/site-packages/ai_models_graphcast/model.py", line 226, in run
    ) = data_utils.extract_inputs_targets_forcings(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/graphcast/data_utils.py", line 354, in extract_inputs_targets_forcings
    inputs = inputs[list(input_variables)]
             ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/xarray/core/dataset.py", line 1552, in __getitem__
    return self._copy_listed(key)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/xarray/core/dataset.py", line 1410, in _copy_listed
    ref_name, var_name, var = _get_virtual_variable(
                              ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/xarray/core/dataset.py", line 207, in _get_virtual_variable
    raise KeyError(key)
KeyError: 'geopotential'

also with fourcastnet;
ai-models --input opendata --lead-time 24 fourcastnet

2024-03-28 14:26:20,758 INFO Writing results to fourcastnet.grib.
2024-03-28 14:26:20,758 INFO Loading ./global_means.npy
2024-03-28 14:26:20,759 INFO Loading ./global_stds.npy
2024-03-28 14:26:20,759 INFO Loading surface fields from OPENDATA
2024-03-28 14:26:20,779 WARNING MARS post-processing keywords {'area'} not supported
2024-03-28 14:26:22,066 INFO Downloading <multiple>
2024-03-28 14:26:28,038 INFO Loading pressure fields from OPENDATA                                                                                                                          
2024-03-28 14:26:28,430 WARNING No index entries for param=z
2024-03-28 14:26:29,010 INFO Downloading <multiple>
2024-03-28 14:26:41,956 INFO Total time: 22 seconds.                                                                                                                                        
Traceback (most recent call last):
  File "/usr/bin/ai-models", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/lib/python3.11/site-packages/ai_models/__main__.py", line 322, in main
    _main(sys.argv[1:])
  File "/usr/lib/python3.11/site-packages/ai_models/__main__.py", line 270, in _main
    run(vars(args), unknownargs)
  File "/usr/lib/python3.11/site-packages/ai_models/__main__.py", line 295, in run
    model.run()
  File "/usr/lib/python3.11/site-packages/ai_models_fourcastnet/model.py", line 159, in run
    all_fields_numpy = self.normalise(all_fields_numpy)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ai_models_fourcastnet/model.py", line 109, in normalise
    new_data = (data - self.means) / self.stds
                ~~~~~^~~~~~~~~~~~
ValueError: operands could not be broadcast together with shapes (1,14,720,1440) (1,26,1,1)

[koldunovn]

Same here:

2024-04-13 09:26:12,382 INFO Writing results to fourcastnet.grib.
2024-04-13 09:26:12,382 INFO Loading ./global_means.npy
2024-04-13 09:26:12,383 INFO Loading ./global_stds.npy
2024-04-13 09:26:16,568 INFO Total time: 6 seconds.
Traceback (most recent call last):
  File "/work/ab0995/a270088/mambaforge/envs/ai-models/bin/ai-models", line 8, in <module>
    sys.exit(main())
  File "/work/ab0995/a270088/mambaforge/envs/ai-models/lib/python3.10/site-packages/ai_models/__main__.py", line 322, in main
    _main(sys.argv[1:])
  File "/work/ab0995/a270088/mambaforge/envs/ai-models/lib/python3.10/site-packages/ai_models/__main__.py", line 270, in _main
    run(vars(args), unknownargs)
  File "/work/ab0995/a270088/mambaforge/envs/ai-models/lib/python3.10/site-packages/ai_models/__main__.py", line 295, in run
    model.run()
  File "/work/ab0995/a270088/mambaforge/envs/ai-models/lib/python3.10/site-packages/ai_models_fourcastnet/model.py", line 159, in run
    all_fields_numpy = self.normalise(all_fields_numpy)
  File "/work/ab0995/a270088/mambaforge/envs/ai-models/lib/python3.10/site-packages/ai_models_fourcastnet/model.py", line 109, in normalise
    new_data = (data - self.means) / self.stds
ValueError: operands could not be broadcast together with shapes (1,16,720,1440) (1,26,1,1)

[Dadoof]

So... was this ever resolved?

[mchantry]

Currently not all the required data for this model is part of the open-data offering. We hope to fix this in the near future, but I cannot provide a timeline at present.

[david5010]

Hi @mchantry,
If that's the case, how is ECMWF running GraphCast here? I'm wondering what initial conditions they're using.

[mrydz]

ECMWF doesn't need to use open-data, they have the full model output available (and licensed users can get all the required inputs via dissemination).