Fix for expver dimension

[DarshanSP19]

Added the fix for expver dimension for recent months data.

[DarshanSP19]

Hey @alxmrs We for the majority of the data files this fix seems working (Verified after back filling the data). But following two variables are hitting the assertion assert disjoint_nans, "The nans are not disjoint in expver=1 vs 5". It seems like data itself is inappropriate.

cloud_base_height
convective_inhibition

[alxmrs]

LGTM. If this has been tested on realistic data, than it makes sense to me.

[alxmrs]

According to Alvaro ([email protected]), the fix should be:

dataarray.sel(expver=1, drop=True)

[alvarosg]

If I understand correctly the reason why this was not working is that dataarray.sel(expver=1) and dataarray.sel(expver=5),
even though they no longer have the "expver" dim, they both remember which expver coordinate they came from (they keep a scalar expver coordinate), and then they complain when running arithmetics between them because the scalar coordinate don't match. Then by adding "expver" to the dimensions to reduce the scalar coordinate diappears too. However this is unintuitive, because the propose code tries to reduce across a dim, which no longer exists.

I believe the right fix, rather than setting:
all_dims_except_time = tuple(set(dataarray.dims) - {"time", "expver"})

should be to use:

dataarray.sel(expver=1, drop=True)
dataarray.sel(expver=5, drop=True)

If you confirm this alternative fix works, this would be preferable.

I think the right fix for this should be:

[alxmrs]

Talking more with Alvaro here: I think these errors only crop up if we download ERA5T, not ERA5. If we simply updated the data, or put limits into how recently we were updating the data, I believe these issues would go away.

[DarshanSP19]

We've downloaded the data for 2023 on 21-22 September. Will that cause any trouble?
Tried below snipped giving an error.

all_dims_except_time = tuple(set(dataarray.dims) - {"time"})
a = dataarray.sel(expver=1, drop=True).isnull().any(dim=all_dims_except_time)
b = dataarray.sel(expver=5, drop=True).isnull().any(dim=all_dims_except_time)
disjoint_nans = bool(next(iter((a ^ b).all().data_vars.values())))
assert disjoint_nans, "The nans are not disjoint in expver=1 vs 5"
dataarray = dataarray.sel(expver=1).combine_first(dataarray.sel(expver=5))

Error

ValueError: 'expver' not found in array dimensions ('time', 'latitude', 'longitude')

[alvarosg]

Could you indicate which line is throwing the error?

Perhaps the problem is that this file has an "expver" coordinate, but not an "expver" dim, which is something we have not found before, but can probably be solved by changing:

if "expver" in dataarray.coords:
   # previous code

to

if "expver" in dataarray.dims:
  # previous code
elif "expver" in dataarray.coords:
  dataarray = dataarray.drop_vars('expver')

[alxmrs]

We've downloaded the data for 2023 on 21-22 September. Will that cause any trouble?

In short, yes I do think this is the root issue. Our goal with keeping data up-to-date is to have a ~1-2 month delay in freshness exactly for this reason. Unless we plan to re-download and update ERA5T data to ERA5 (I don't recommend this), then it would be best to change the interval at which we ingest data to avoid the case that this PR is trying to solve for.

[DarshanSP19]

@alxmrs After verifying the code for April, May and June 2023 data, It seems that the changes are required for data ingestion. As the data for April and May also downloaded on 21st September so it should be fresh enough for the process further.

[alxmrs]

LGTM.

[alxmrs]

Thanks for the discussion in this PR.