You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Totally amazing project. Wanted to ask about compression/uint16 storage. When I get lat/lon ERA5 data from the CDS api I see the data is compressed and stored in uint16 along with scale factors to convert back to float32. When I look at the dataset here it seems to be in the full float32. Any reason not to store in the original uint16? Saves tremendous amounts of bandwidth and when using ERA5 in ML workflows this can make a big difference.
The text was updated successfully, but these errors were encountered:
This is definitely worth investigating. I know that we didn't do this for some 3D fields because the scale/offset factors that ECMWF uses differ for the same variable at different vertical levels in the atmosphere.
Hey,
Totally amazing project. Wanted to ask about compression/uint16 storage. When I get lat/lon ERA5 data from the CDS api I see the data is compressed and stored in uint16 along with scale factors to convert back to float32. When I look at the dataset here it seems to be in the full float32. Any reason not to store in the original uint16? Saves tremendous amounts of bandwidth and when using ERA5 in ML workflows this can make a big difference.
The text was updated successfully, but these errors were encountered: